mlsanityData to 7 Oct 2026

MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

arXiv 2605.22949 ↗First posted 21 May 2026cs.LG cs.MAcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 35 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

21 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Large Language Models Miss the Multi-Agent Mark27 May
  2. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 19

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
7 more
  1. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  2. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  3. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  4. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  5. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  6. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  7. 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibratorcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
  2. 2026From Migration to Calibration: Preserving Agent Capabilities across Models, Jurisdictions, and Scale
  3. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation
  4. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
  5. 2026Agentic Confidence Calibration
  6. 2026Calibration-risk routing for controlled world-model adaptation
  7. 2026Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration
  8. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling