mlsanityData to 7 Oct 2026

GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning

arXiv 2504.02546 ↗First posted 3 Apr 2025cs.LG cs.AIcited by 147

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) can directly enhance the reasoning capabilities of large language models without extensive reliance on Supervised Fine-Tuning (SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimize the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KL divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. As illustrated in Figure 1, extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks. Our code is available at https://github.com/AMAP-ML/GPG.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 65 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

31 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
10 more
  1. 2025LIMO: Less is More for Reasoningcited by 488
  2. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  3. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  4. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  5. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  6. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  8. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
  9. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  10. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 76; 2026: 71.

147 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  4. 2025Geometric-Mean Policy Optimizationcited by 107
  5. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  6. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  7. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  8. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  9. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  10. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  11. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  12. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 50
38 more
  1. 2025ExGRPO: Learning to Reason from Experiencecited by 49
  2. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  3. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  4. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  5. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
  6. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  7. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 35
  8. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 33
  9. 2025AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Drivingcited by 32
  10. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  11. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  12. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  13. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  14. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
  15. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 25
  16. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  17. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
  18. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  19. 2025FingER: Content Aware Fine-grained Evaluation with Reasoning for AI-Generated Videoscited by 22
  20. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  21. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  22. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  23. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  24. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  25. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  26. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulationcited by 18
  27. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
  28. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  29. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  30. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  31. 2025Play to Generalize: Learning to Reason Through Game Playcited by 15
  32. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
  33. 2026NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoningcited by 15
  34. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  35. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 14
  36. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimizationcited by 14
  37. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolutioncited by 14
  38. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
  2. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  3. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
  4. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  5. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  6. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
  7. 2026GRPO Training Dynamics for Small Language Models
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning