mlsanityData to 14 Oct 2026

VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

arXiv 2504.05118 ↗First posted 7 Apr 2025cs.AIcited by 285

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We present VAPO, Value-based Augmented Proximal Policy Optimization framework for reasoning models., a novel framework tailored for reasoning models within the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the Qwen 32B pre-trained model, attains a state-of-the-art score of 60.4. In direct comparison under identical experimental settings, VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B and DAPO by more than 10 points. The training process of VAPO stands out for its stability and efficiency. It reaches state-of-the-art performance within a mere 5,000 steps. Moreover, across multiple independent runs, no training crashes occur, underscoring its reliability. This research delves into long chain-of-thought (long-CoT) reasoning using a value-based reinforcement learning framework. We pinpoint three key challenges that plague value-based methods: value model bias, the presence of heterogeneous sequence lengths, and the sparsity of reward signals. Through systematic design, VAPO offers an integrated solution that effectively alleviates these challenges, enabling enhanced performance in long-CoT reasoning tasks.

Key facts. Cited by 285 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 61 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

14 of 61 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  5. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 412
  6. 2019F3Net: Fusion, Feedback and Focus for Salient Object Detectioncited by 176
  7. 2023LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusioncited by 58
  8. 2022ShuffleMixer: An Efficient ConvNet for Image Super-Resolutioncited by 45
  9. 2022FIFO: Learning Fog-invariant Features for Foggy Scene Segmentationcited by 31
  10. 2021Deep Instance Segmentation with Automotive Radar Detection Pointscited by 22
  11. 2025LXLv2: Enhanced LiDAR Excluded Lean 3D Object Detection with Fusion of 4D Radar and Cameracited by 13
  12. 2023ParameterNet: Parameters Are All You Needcited by 7
2 more
  1. 2024WaveDH: Wavelet Sub-bands Guided ConvNet for Efficient Image Dehazingcited by 3
  2. 2024AGSENet: A Robust Road Ponding Detection Method for Proactive Traffic Safetycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 158; 2026: 127.

285 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  3. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  6. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  7. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025Agentic Reinforced Policy Optimizationcited by 172
  11. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  12. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
38 more
  1. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  2. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  3. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  4. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  5. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  6. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  7. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 85
  8. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  9. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  10. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  11. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  12. 2025Reinforcement Learning from Human Feedbackcited by 58
  13. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  14. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  15. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  16. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  17. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  18. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 52
  19. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  20. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  21. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  22. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  23. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 43
  24. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  25. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  26. 2025DCPO: Dynamic Clipping Policy Optimizationcited by 40
  27. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 39
  28. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  29. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 39
  30. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  31. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  32. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  33. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  34. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  35. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 34
  36. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  37. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 32
  38. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 32

50 most cited of 285 All 285 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
  2. 2025Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
  3. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning
  4. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
  5. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
  6. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
  7. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimization
  8. 2026GraphPO: Graph-based Policy Optimization for Reasoning Models