mlsanityData to 14 Oct 2026

Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning

arXiv 2503.15952 ↗First posted 20 Mar 2025cs.CLcited by 21

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which uses a simple but effective method, an adaptive loss function, to mitigate training fluctuation and token inefficiency. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 21 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

15 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  10. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 96
  11. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025Efficient Reasoning Models: A Surveycited by 95
  2. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 54
  3. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  4. 2025Think When You Need: Self-Adaptive Chain-of-Thought Learningcited by 38
  5. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  6. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  7. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Modelscited by 13
  8. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 13
  9. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  10. 2025DeepPrune: Parallel Scaling without Inter-trace Redundancycited by 10
  11. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 9
  12. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
9 more
  1. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  2. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  3. 2025Enhancing Large Language Models through Structured Reasoningcited by 5
  4. 2025ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Codecited by 2
  5. 2026DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulationcited by 2
  6. 2026Prefix Sliding for efficient test-time scalingcited by 2
  7. 2026A State-Transition Framework for Efficient LLM Reasoningcited by 1
  8. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  9. 2025GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  2. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
  3. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  4. 2026Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization
  5. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  6. 2026Difficulty-Estimated Policy Optimization
  7. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  8. 2026AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback