mlsanityData to 14 Oct 2026

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

arXiv 2309.15028 ↗First posted 26 Sep 2023cs.CL cs.AI cs.LGcited by 60

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Inference-time search algorithms such as Monte-Carlo Tree Search (MCTS) may seem unnecessary when generating natural language text based on state-of-the-art reinforcement learning such as Proximal Policy Optimization (PPO). In this paper, we demonstrate that it is possible to get extra mileage out of PPO by integrating MCTS on top. The key idea is not to throw out the value network, a byproduct of PPO training for evaluating partial output sequences, when decoding text out of the policy network. More concretely, we present a novel value-guided decoding algorithm called PPO-MCTS, which can integrate the value network from PPO to work closely with the policy network during inference-time generation. Compared to prior approaches based on MCTS for controlled text generation, the key strength of our approach is to reduce the fundamental mismatch of the scoring mechanisms of the partial outputs between training and test. Evaluation on four text generation tasks demonstrate that PPO-MCTS greatly improves the preferability of generated text compared to the standard practice of using only the PPO policy. Our results demonstrate the promise of search algorithms even on top of the aligned language models from PPO, and the under-explored benefit of the value network.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 51 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

36 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
20 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  2. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  4. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  7. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  8. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  9. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  10. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  11. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  12. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  13. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  14. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 149
  15. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  16. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  17. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  18. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  19. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  20. 2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 39; 2026: 10.

60 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,341
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  4. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
  5. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  6. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  7. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  8. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  9. 2025Reinforcement Learning from Human Feedbackcited by 58
  10. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  11. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  12. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 45
38 more
  1. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  2. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  3. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  5. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 24
  6. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  7. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 22
  8. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectivenesscited by 20
  9. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  10. 2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
  11. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14
  12. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 13
  13. 2025CoAT: Chain-of-Associated-Thoughts Framework for Enhancing Large Language Models Reasoningcited by 12
  14. 2025On the Query Complexity of Verifier-Assisted Language Generationcited by 12
  15. 2025First Finish Search: Efficient Test-Time Scaling in Large Language Modelscited by 9
  16. 2025Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Modelscited by 9
  17. 2025A Probabilistic Inference Scaling Theory for LLM Self-Correctioncited by 9
  18. 2026Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scalingcited by 9
  19. 2026UniT: Unified Multimodal Chain-of-Thought Test-time Scalingcited by 8
  20. 2025When Can Model-Free Reinforcement Learning be Enough for Thinking?cited by 6
  21. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  22. 2026Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Modelcited by 6
  23. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Taskcited by 5
  24. 2025It's Not That Simple. An Analysis of Simple Test-Time Scalingcited by 5
  25. 2024Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learningcited by 4
  26. 2025Language Models can Self-Improve at State-Value Estimation for Better Searchcited by 4
  27. 2025ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Callingcited by 4
  28. 2025COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Modelscited by 3
  29. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
  30. 2025Streaming Looking Ahead with Token-level Self-rewardcited by 2
  31. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Trainingcited by 2
  32. 2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
  33. 2025Monte Carlo Tree Diffusion with Multiple Experts for Protein Designcited by 2
  34. 2025Hierarchical Vision Language Action Model Using Success and Failure Demonstrationscited by 2
  35. 2026Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agentscited by 2
  36. 2026Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scalingcited by 2
  37. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  38. 2025Breaking the Myth: Can Small Models Infer Postconditions Too?cited by 1

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TPPO: Tracking-Value PPO For Language Model Reasoning
  2. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  3. 2025I-MCTS: Enhancing Agentic AutoML via Introspective Monte Carlo Tree Search
  4. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
  5. 2026Consensus Group Relative Policy Optimization for Text Generation
  6. 2024A Critical Look At Tokenwise Reward-Guided Text Generation
  7. 2022Critic-Guided Decoding for Controlled Text Generation
  8. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks