# One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

arXiv: https://arxiv.org/abs/2607.28317 · HTML: https://mlsanity.com/paper/2607.28317/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 18 of 18)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [The Flaws of Policies Requiring Human Oversight of Government Algorithms](https://mlsanity.com/paper/2109.05067/) (2021)
- [Principled Approaches for Learning to Defer with Multiple Experts](https://mlsanity.com/paper/2310.14774/) (2023)
- [Multi-Armed Bandits with Correlated Arms](https://mlsanity.com/paper/1911.03959/) (2019)
- [Batch Multi-Fidelity Active Learning with Budget Constraints](https://mlsanity.com/paper/2210.12704/) (2022)
- [The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents](https://mlsanity.com/paper/2601.07264/) (2026)
- [Agentic Confidence Calibration](https://mlsanity.com/paper/2601.15778/) (2026)
- [A Causal Framework for Evaluating Deferring Systems](https://mlsanity.com/paper/2405.18902/) (2024)
- [Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints](https://mlsanity.com/paper/2403.06906/) (2024)
- [Correlated Multi-armed Bandits with a Latent Random Source](https://mlsanity.com/paper/1808.05904/) (2018)
- [Governing What the EU AI Act Excludes: Accountability for Autonomous AI Agents in Smart City Critical Infrastructure](https://mlsanity.com/paper/2605.01091/) (2026)

## Abstract (arXiv)

A single human must audit N LLM agents under a budget of B ll N audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold δ^* past which confidence-ranked auditing is worse than random. Two a-priori expectations reverse: δ^* rises as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for vacuous oversight, and replaying policies on recorded traces confirms the ordering.