COVER: Identifiable Evaluation of Coalition Routing
Not on the map. Why?
Abstract (arXiv)
When a multi-agent system changes its team, it also changes the messages and final answer it produces, so an end-to-end accuracy gap does not by itself identify a routing effect. We introduce method, an evaluation contract that fixes a public information boundary, downstream stack G, and finite legal team family before outcomes are generated. Complete coverage identifies exact finite-benchmark oracle regret conditional on that stack. For any finite collection of frozen policies, executing the union of their distinct selected teams is the minimal assumption-free support for every pairwise policy contrast, though not for absolute oracle regret. Two controlled tables with source-ID-disjoint splits test the instrument. On MuSiQue-12, a pre-specified privileged positive control improves regret from 0.532 to 0.402; a later public-interface control reaches 0.424 versus 0.554 but is retrospective. On HotpotQA-4, a pre-specified public direct scorer improves regret from 0.313 to 0.110. In fixed-stack Llama execution, verified route regret improves by 0.190, while the raw-answer gain is 0.010 with an interval crossing zero. A five-family ToolSandbox variant-shift validation exhaustively evaluates 16 declared teams on 14 untouched task variants (224/224 valid rows): the declared-family oracle reaches 0.768 safe-evidence completion, while the prospectively frozen router gets 0.637 (regret 0.131), failing the predeclared 0.10 criterion. A later retrospective comparator reaches 0.655, matching all-workers with 4.57 versus 5.00 workers on average. Thus COVER exposes selection headroom without manufacturing a routing win. A crossed-stack diagnostic shows absolute scores depend on G but finds no detectable router-by-finalizer interaction. COVER is an auditable measurement methodology, not a claim of stack-invariant or universal agent-routing superiority.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 20 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
10 of 20 references.
References · 10
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2017Deep Setscited by 1,421
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
- 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
- 2024Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Banditscited by 8
- 2023Leveraging Factored Action Spaces for Off-Policy Evaluationcited by 3
- 2026Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimationcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization
- 2026What AI Red-Team Evaluations Can and Cannot Prove
- 2026RouteGuard: Certifying Routing Gain in LLM Multi-Agent Systems When Complementarity Is Not Enough
- 2026Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing
- 2026Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?
- 2026RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
- 2025MASPRM: Multi-Agent System Process Reward Model
- 2026Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment