mlsanityData to 7 Oct 2026

VinePPO: Refining Credit Assignment in RL Training of LLMs

arXiv 2410.01679 ↗First posted 2 Oct 2024cs.LG cs.CLcited by 168

Not on the map. Why?

Cited by 104 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model performance. Proximal Policy Optimization (PPO), a common reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, recent approaches achieve strong results without it, raising questions about the efficacy of value networks in practice. In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they often produce poor estimate of expected return and barely outperform a random baseline when comparing alternative steps. This motivates our key question: Can improved credit assignment enhance RL training for LLMs? To address this, we propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates. Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time (up to 3.0x). Crucially, it achieves higher test accuracy for a given training accuracy, capturing more generalization signal per sample. These results emphasize the importance of accurate credit assignment in RL training of LLM.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

35 of 46 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
23 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  6. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  8. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  9. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  10. 2024LoRA Learns Less and Forgets Lesscited by 313
  11. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  13. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  14. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  15. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  16. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  17. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  18. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  19. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  20. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  21. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  22. 2023Learning to Generate Better Than Your LLMcited by 49
  23. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 62; 2026: 102.

168 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2026Reinforcement Learning via Self-Distillationcited by 285
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  8. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  9. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  12. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
38 more
  1. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  3. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  4. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  5. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  6. 2025Reinforcement Learning from Human Feedbackcited by 53
  7. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  9. 2025TreeRPO: Tree Relative Policy Optimizationcited by 45
  10. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  11. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  12. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
  13. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  14. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  15. 2025Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Propertiescited by 28
  16. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agentscited by 28
  17. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  18. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  19. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  20. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  21. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  22. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  23. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  24. 2025GEM: A Gym for Agentic LLMscited by 17
  25. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  26. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  27. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  28. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  29. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  30. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14
  31. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
  32. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  33. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoningcited by 13
  34. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  35. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  36. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
  37. 2025AdapThink: Adaptive Thinking Preferences for Reasoning Language Modelcited by 11
  38. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 11

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  3. 2025Token-Efficient RL for LLM Reasoning
  4. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
  6. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  7. 2026VIMPO: Value-Implicit Policy Optimization for LLMs
  8. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning