mlsanityData to 14 Oct 2026

VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training

arXiv 2508.03058 ↗First posted 5 Aug 2025cs.LG cs.AI cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete reward supervision, which undermines policy stability and generalization. Such noise may cause models to ignore key information or even collapse in advantage estimation. We find that a strong value model is essential for absorbing unstable signals and producing reliable advantages, offering denser and more robust supervision than the reward model. To better optimize noisy supervision, we propose VRPO, a framework that enhances value modeling for robust RL in LLM post-training. VRPO integrates (1) auxiliary losses guided by entropy and perplexity from a frozen language model, and (2) a variational information bottleneck, enabling the value model to filter noise and capture key words. This design allows the value model to correct noise rewards and generate more reliable advantage estimates, transforming it from a passive predictor into an active noise regulator. Experiments on multi-turn dialogue, math reasoning, and science QA with both rule-based and model-based rewards show that VRPO consistently outperforms baselines such as PPO and GRPO. Our work highlight the central role of the value model in Robust RL and provide a principled and practical approach to policy optimization under noisy supervision.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

34 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  2. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning22 May

Other references · 32

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2025Humanity's Last Examcited by 491
  11. 2024Measuring short-form factuality in large language modelscited by 373
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
20 more
  1. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  3. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  6. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  7. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  8. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  9. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  10. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  11. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  12. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  13. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  14. 2022MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspectivecited by 17
  15. 2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
  16. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  17. 2024SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Modelscited by 10
  18. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
  19. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
  20. 2023Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  3. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  4. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  5. 2026SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoningcited by 1
  6. 2026IACM-RL: Intent-Aware Context Management and Reinforcement Learning for Complex Tool Invocation under Dynamic Intent Fluctuationscited by 1
  7. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Trainingcited by 0
  8. 2026FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learningcited by 0
  9. 2026Explicit Critic Guidance for Aligning Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
  2. 2025Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
  3. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
  4. 2026Value-Gradient Hypothesis of RL for LLMs
  5. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
  6. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  7. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
  8. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks