mlsanityData to 7 Oct 2026

Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols

arXiv 2605.09618 ↗First posted 10 May 2026cs.CL cs.CYcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals? We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner. This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold. The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 3 of its 9 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

3 of 9 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning7 Apr

Other references · 2

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
  2. 2026ARMOR-MAD: Adaptive Routing for Heterogeneous Multi-Agent Debate in Large Language Model Reasoning
  3. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
  4. 2025Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
  5. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
  6. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  7. 2025From Belief Entrenchment to Robust Reasoning in LLM Agents
  8. 2025Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate