Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70pp and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector's 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67pp, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model's pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 33 references that are in the map. First posted on arXiv 20 Jul 2026.
Builds on
32 of 33 references.
Closest ancestors · 9
Same direction, newest first
- 2026LLM-as-a-Verifier: A General-Purpose Verification Framework6 Jul
- 2026Parallel Test-Time Scaling with Multi-Sequence Verifiers3 Mar
- 2025Trust but Verify! A Survey on Verification Design for Test-time Scaling20 Aug
- 2025ROC-n-reroll: How verifier imperfection affects test-time scaling16 Jul
- 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning1 Apr
- 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers27 Feb
- 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal17 Feb
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
- 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
Other references · 23
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023Let's Verify Step by Stepcited by 3,598
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
- 2022Solving math word problems with process- and outcome-based feedbackcited by 871
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
11 more
- 2022CodeT: Code Generation with Generated Testscited by 369
- 2025Towards a Science of Scaling Agent Systemscited by 125
- 2022Natural Language to Code Translation with Executioncited by 122
- 2024The Limits of Inference Scaling Through Resamplingcited by 51
- 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 28
- 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversitycited by 25
- 2025Learning to Generate Unit Tests for Automated Debuggingcited by 16
- 2025Learning to Generate Unit Test via Adversarial Reinforcement Learningcited by 11
- 2025When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifierscited by 11
- 2026Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoningcited by 5
- 2026CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Testcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning
- 2026Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
- 2026VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
- 2026Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines
- 2026Self-Verification Dilemma: Experience-Driven Suppression of Overused Checking in LLM Reasoning
- 2026LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks
- 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?
- 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning