mlsanityData to 14 Oct 2026

Value-Guided Search for Efficient Chain-of-Thought Reasoning

arXiv 2505.17373 ↗First posted 23 May 2025cs.LG cs.AI cs.CLcited by 13

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 31 of its 53 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

31 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals18 Apr
  2. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling10 Feb
  3. 2025Competitive Programming with Large Reasoning Models3 Feb
  4. 2025s1: Simple test-time scaling31 Jan

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  12. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
15 more
  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  2. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  3. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  4. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  5. 2025Phi-4-reasoning Technical Reportcited by 151
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  7. 2023Controlled Decoding from Language Modelscited by 135
  8. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
  9. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 60
  10. 2023Learning to Generate Better Than Your LLMcited by 49
  11. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  12. 2024Entropy-Regularized Process Reward Modelcited by 38
  13. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  14. 2024Investigating the Histogram Loss in Regressioncited by 11
  15. 2024Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  3. 2025Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughtscited by 12
  4. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 12
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  6. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  7. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  8. 2026Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inferencecited by 3
  9. 2026Believe Your Model: Distribution-Guided Confidence Calibrationcited by 2
  10. 2026The Power of Test-Time Training for Approximate Samplingcited by 1
  11. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  12. 2026Caterpillar of Thoughts: The Optimal Test-Time Algorithm for Large Language Modelscited by 0
1 more
  1. 2026Improving Math Reasoning through Value-guided Informative Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Language Models can Self-Improve at State-Value Estimation for Better Search
  2. 2026Learning to Reason Efficiently with A* Post-Training
  3. 2025DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
  4. 2025Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
  5. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
  6. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
  7. 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
  8. 2026APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards