mlsanityData to 14 Oct 2026

Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute

arXiv 2504.00762 ↗First posted 1 Apr 2025cs.AIcited by 20

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

This paper presents a simple, effective, and cost-efficient strategy to improve LLM performance by scaling test-time compute. Our strategy builds upon the repeated-sampling-then-voting framework, with a novel twist: incorporating multiple models, even weaker ones, to leverage their complementary strengths that potentially arise from diverse training data and paradigms. By using consistency as a signal, our strategy dynamically switches between models. Theoretical analysis highlights the efficiency and performance advantages of our strategy. Extensive experiments on six datasets demonstrate that our strategy not only outperforms self-consistency and state-of-the-art multi-agent debate approaches, but also significantly reduces inference costs. Additionally, ModelSwitch requires only a few comparable LLMs to achieve optimal performance and can be extended with verification methods, demonstrating the potential of leveraging multiple LLMs in the generation-verification paradigm.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 59 of its 66 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

59 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers27 Feb
  2. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling10 Feb
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  4. 2023Universal Self-Consistency for Large Language Model Generation29 Nov

Other references · 55

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  13. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  14. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  15. 2025Process Reinforcement through Implicit Rewardscited by 504
  16. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  17. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  18. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  19. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  20. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  21. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  22. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  23. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  24. 2024More Agents Is All You Needcited by 180
  25. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  26. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  27. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  28. 2024To Believe or Not to Believe Your LLMcited by 120
  29. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114
  30. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 107
  31. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  32. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 105
  33. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  34. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  35. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  36. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  37. 2021Lower Perplexity is Not Always Human-Likecited by 52
  38. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 9.

20 papers.

Sort:

By area:

  1. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 44
  2. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  3. 2026Inference-Time Budget Control for LLM Search Agentscited by 13
  4. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  5. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 10
  6. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 10
  7. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  8. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
  9. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 7
  10. 2025Making, not Taking, the Best of Ncited by 6
  11. 2026Cross-Model Disagreement as a Label-Free Correctness Signalcited by 6
  12. 2025Rethinking Optimal Verification Granularity for Compute-Efficient Test-Time Scalingcited by 5
8 more
  1. 2026ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generationcited by 5
  2. 2025TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendationcited by 3
  3. 2025Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMscited by 3
  4. 2026VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selectioncited by 2
  5. 2025FEval-TTC: Fair Evaluation Protocol for Test-Time Computecited by 1
  6. 2026Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scalecited by 1
  7. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 1
  8. 2026More Correct Mass, Worse Answers: Why Power Sampling Can Fail and How to Fix Itcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
  2. 2026Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference
  3. 2025RoBoN: Routed Online Best-of-n for Test-Time Scaling with Multiple LLMs
  4. 2024A Survey on LLM Inference-Time Self-Improvement
  5. 2025Scaling Test-time Compute for LLM Agents
  6. 2025A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
  7. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal
  8. 2026Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning