# Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning

arXiv: https://arxiv.org/abs/2503.15952 · HTML: https://mlsanity.com/paper/2503.15952/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models](https://mlsanity.com/paper/2410.09671/) (2024)

## Built on by (most cited first, 21 of 21)

- [Efficient Reasoning Models: A Survey](https://mlsanity.com/paper/2504.10903/) (2025)
- [R-KV: Redundancy-aware KV Cache Compression for Reasoning Models](https://mlsanity.com/paper/2505.24133/) (2025)
- [Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards](https://mlsanity.com/paper/2506.00103/) (2025)
- [Think When You Need: Self-Adaptive Chain-of-Thought Learning](https://mlsanity.com/paper/2504.03234/) (2025)
- [OctoNav: Towards Generalist Embodied Navigation](https://mlsanity.com/paper/2506.09839/) (2025)
- [VeriThinker: Learning to Verify Makes Reasoning Model Efficient](https://mlsanity.com/paper/2505.17941/) (2025)
- [Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models](https://mlsanity.com/paper/2505.17697/) (2025)
- [The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning](https://mlsanity.com/paper/2510.06557/) (2025)
- [G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance](https://mlsanity.com/paper/2508.13023/) (2025)
- [DeepPrune: Parallel Scaling without Inter-trace Redundancy](https://mlsanity.com/paper/2510.08483/) (2025)
- [SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2604.08865/) (2026)
- [C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning](https://mlsanity.com/paper/2509.23129/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation](https://mlsanity.com/paper/2605.21125/) (2026)
- [Enhancing Large Language Models through Structured Reasoning](https://mlsanity.com/paper/2506.20241/) (2025)
- [ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code](https://mlsanity.com/paper/2509.07006/) (2025)
- [DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation](https://mlsanity.com/paper/2607.13033/) (2026)
- [Prefix Sliding for efficient test-time scaling](https://mlsanity.com/paper/2608.26070/) (2026)
- [A State-Transition Framework for Efficient LLM Reasoning](https://mlsanity.com/paper/2602.01198/) (2026)
- [A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions](https://mlsanity.com/paper/2606.16733/) (2026)
- [GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoning](https://mlsanity.com/paper/2510.01165/) (2025)

## Abstract (arXiv)

Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which uses a simple but effective method, an adaptive loss function, to mitigate training fluctuation and token inefficiency. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.