mlsanityData to 7 Oct 2026

One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

arXiv 2607.28317 ↗First posted 30 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

A single human must audit N LLM agents under a budget of B ll N audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold δ^* past which confidence-ranked auditing is worse than random. Two a-priori expectations reverse: δ^* rises as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for vacuous oversight, and replaying policies on recorded traces confirms the ordering.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 18 of its 30 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

18 of 30 references.

References · 18

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  5. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  6. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  7. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  8. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  9. 2021The Flaws of Policies Requiring Human Oversight of Government Algorithmscited by 44
  10. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  11. 2019Multi-Armed Bandits with Correlated Armscited by 33
  12. 2022Batch Multi-Fidelity Active Learning with Budget Constraintscited by 20
6 more
  1. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  2. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  3. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  4. 2026Agentic Confidence Calibrationcited by 12
  5. 2018Correlated Multi-armed Bandits with a Latent Random Sourcecited by 6
  6. 2026Governing What the EU AI Act Excludes: Accountability for Autonomous AI Agents in Smart City Critical Infrastructurecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk
  2. 2026Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
  3. 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
  4. 2026The Confidence Game: Strategic Miscalibration in Human-AI Delegation
  5. 2026Calibration Is Not Control: Why LLM-Agent Oversight Needs Intervention
  6. 2026The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines
  7. 2026Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities
  8. 2026Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing