mlsanityData to 7 Oct 2026

REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

arXiv 2501.03262 ↗First posted 4 Jan 2025cs.CL cs.LGcited by 233

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback~(RLHF) plays a crucial role in aligning Large Language Models~(LLMs). The dominant algorithm, Proximal Policy Optimization~(PPO), employs a critic network to estimate advantages, which introduces significant computational and memory overhead. To address this, a family of critic-free algorithms (e.g., GRPO, RLOO) has emerged. However, these methods typically rely on textitprompt-level (local) advantage normalization, which suffers from inaccurate advantage estimation, a tendency to overfit, and, as we show, is a theoretically biased estimator. To solve these challenges, we introduce REINFORCE++, a critic-free framework centered on Global Advantage Normalization. By normalizing advantages across the entire global batch rather than small, prompt-specific groups, our method provides a more stable and theoretically sound, textiteffectively unbiased estimate (whose bias vanishes as batch size increases). We introduce two variants: REINFORCE++, a highly efficient and general algorithm (k ≥ 1) for general-domain RLHF, and REINFORCE++ /w baseline, a robust group-sampling variant (k > 1) for complex reasoning tasks. Our empirical evaluation demonstrates that each variant shows superior stability and performance in its respective domain, outperforming existing methods and even PPO in complex agentic settings.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 25 references that are in the map. First posted on arXiv 4 Jan 2025.

Builds on

7 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 5

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 102; 2026: 131.

233 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  5. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  6. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learningcited by 67
  7. 2025Can Large Reasoning Models Self-Train?cited by 66
  8. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  9. 2025Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learningcited by 50
  10. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  11. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  12. 2025VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thoughtcited by 47
38 more
  1. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 46
  2. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  3. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  4. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  5. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 37
  6. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agentscited by 37
  7. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 35
  8. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 34
  9. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinkingcited by 34
  10. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  11. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  12. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  13. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  14. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  15. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  16. 2025Promoting Efficient Reasoning with Verifiable Stepwise Rewardcited by 27
  17. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  18. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  19. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  20. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  21. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  22. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 20
  23. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Modelscited by 18
  24. 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Modelscited by 17
  25. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  26. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  27. 2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineeringcited by 16
  28. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 16
  29. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learningcited by 16
  30. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  31. 2025R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learningcited by 15
  32. 2025Reasoning with Reinforced Functional Token Tuningcited by 14
  33. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  34. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Modelscited by 14
  35. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  36. 2025Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learningcited by 13
  37. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 13
  38. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment
  2. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  3. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
  4. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  5. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
  6. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
  7. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  8. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients