Data to 14 Oct 2026

Scaling Test-Time Compute Without Verification or RL is Suboptimal

arXiv 2502.12118 ↗First posted 17 Feb 2025cs.LG cs.CLcited by 84

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Despite substantial advances in scaling test-time compute, an ongoing debate in the community is how it should be scaled up to enable continued and efficient improvements with scaling. There are largely two approaches: first, distilling successful search or thinking traces; and second, using verification (e.g., 0/1 outcome rewards, reward models, or verifiers) to guide reinforcement learning (RL) and search algorithms. In this paper, we prove that finetuning LLMs with verifier-based (VB) methods based on RL or search is far superior to verifier-free (VF) approaches based on distilling or cloning search traces, given a fixed amount of compute/data budget. Further, we show that as we scale test-time compute (measured as the output token length) and training data, suboptimality of VF methods scales poorly compared to VB when the base pre-trained LLM presents a heterogeneous distribution over correct solution traces (e.g., different lengths, styles, etc.) and admits a non-sharp distribution over rewards on traces sampled from it. We formalize this condition using anti-concentration [ErdHos, 1945]. This implies a stronger result that VB methods scale better asymptotically, with the performance gap between VB and VF methods widening as test-time budget grows. We corroborate our theory empirically on both didactic and math reasoning problems with 3/8/32B-sized pre-trained LLMs, where we find verification is crucial for scaling test-time compute.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 77 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

50 of 77 references.

Closest ancestors · 10

Same direction, newest first

Other references · 40

2024The Llama 3 Herd of Modelscited by 14,715
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
2022Training Compute-Optimal Large Language Modelscited by 3,019
2024Qwen2 Technical Reportcited by 2,120
2022Solving math word problems with process- and outcome-based feedbackcited by 901
2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
2025Process Reinforcement through Implicit Rewardscited by 504
2025LIMO: Less is More for Reasoningcited by 498
2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480

28 more

2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
2025Training Language Models to Reason Efficientlycited by 247
2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
2024Free Process Rewards without Process Labelscited by 165
2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
2024Generative Reward Modelscited by 148
2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
2021Scaling Scaling Laws with Board Gamescited by 63
2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
2024Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systemscited by 25
2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
2024Online Estimation via Offline Estimation: An Information-Theoretic Frameworkcited by 13
2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Boundscited by 11
2024Learning to Better Search with Language Models via Guided Reinforced Self-Trainingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 59; 2026: 25.

84 papers.

Sort:

By area:

2025Why Do Multi-Agent LLM Systems Fail?cited by 604
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
2025Learning to Reason under Off-Policy Guidancecited by 281
2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 124
2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
2025Inducing Programmatic Skills for Agentic Taskscited by 111
2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109

38 more

2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 53
2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 42
2025RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correctioncited by 39
2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
2025Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scalingcited by 30
2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 25
2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 19
2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14
2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcementcited by 14
2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 14
2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selectioncited by 13
2025Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Onescited by 13
2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
2026Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RLcited by 12
2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
2025Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Modelscited by 11
2025VerifiAgent: a Unified Verification Agent in Language Model Reasoningcited by 10
2025Faster and Better LLMs via Latency-Aware Test-Time Scalingcited by 9
2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 9
2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8
2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamicscited by 8
2026Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolutioncited by 8
2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Explorationcited by 7
2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 6
2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluationcited by 6
2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
2025On the Role of Feedback in Test-Time Scaling of Agentic AI Workflowscited by 4

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way