mlsanityData to 14 Oct 2026

Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence

arXiv 2608.12895 ↗First posted 13 Aug 2026cs.AI cs.MAcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Compositional reliability bounds for multi-agent systems multiply component reliabilities, a step licensed by a conditional-independence assumption that is routinely stated and rarely tested. We test it. Two instances of one model, in a two-agent handoff, co-fail on 90.0% of the missions on which either fails (log OR 6.66, 95% CI [6.38, 7.00]; phi 0.916), in a preregistered evaluation of 18,000 missions scored by deterministic code with no LLM judge. Substituting a different model reduces the association in six of six contrasts; substituting a different vendor, model already different, does not -- a registered hypothesis reported as a null. The error is signed and runs against the operator: positive dependence inflates joint failure above the independence product, so redundancy is over-credited exactly when components share a model. The assumption-free alternative is often vacuous, and fitting a dependence model is worse: we prove a bootstrap bound on a fitted model's functional loses coverage of the truth as n grows, the identification gap being O(1) while the bootstrap haircut is O(n^{-1/2}). More data makes such a certificate worse, with no visible symptom. We give a finite-sample certificate assuming no dependence structure: a linear program over the joint, over a Bonferroni-Clopper-Pearson box around measured co-execution moments. It is sound, sharp for the information supplied, and monotone in the moment family. Enriching ten moment functionals to fourteen narrows the identified interval by 85.7% and lifts the certified floor from 0.2455 to 0.4116. A companion anytime-valid certificate holds type-I error at 0.0471 under optional stopping. Common dependence statistics are marginal-bounded and can reverse an apparent ordering of conditions when the compared agents fail at different rates. Contracts, scoring code, analysis scripts, and the preregistration are released.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 64 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

28 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Counterfactual Graph for Multi-Agent LLM Calibration28 May
  2. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerance13 Nov
  3. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  11. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  12. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
12 more
  1. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  2. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
  3. 2020Estimating means of bounded random variables by bettingcited by 169
  4. 2025Multi-Agent Risks from Advanced AIcited by 163
  5. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
  6. 2019Safe Testingcited by 101
  7. 2026International AI Safety Report 2026cited by 22
  8. 2026Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agentscited by 19
  9. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Searchcited by 12
  10. 2026Can AI Agents Agree?cited by 7
  11. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systemscited by 5
  12. 2026Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triagecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems
  2. 2026Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems
  3. 2026Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures
  4. 2026Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agents
  5. 2026Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment
  6. 2026ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures
  7. 2026When Should Agent Trust Be Conditional? Characterizing and Attacking Skill-Conditional Reputation in Agent Swarms
  8. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation