mlsanityData to 13 Oct 2026

Geometric-Mean Policy Optimization

arXiv 2507.20673 ↗First posted 28 Jul 2025cs.CLcited by 107

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has significantly enhanced the reasoning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unfortunately, GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards, which manifest as extreme importance sampling ratios during training. In this study, we propose Geometric-Mean Policy Optimization (GMPO), with the aim to improve the stability of GRPO through suppressing token reward outliers. Instead of optimizing the arithmetic mean, GMPO maximizes the geometric mean of token-level rewards, which is inherently less sensitive to outliers and maintains a more stable range of importance sampling ratio. GMPO is plug-and-play-simply replacing GRPO's arithmetic mean with the geometric mean of token-level rewards, as the latter is inherently less sensitive to outliers. GMPO is theoretically plausible-analysis reveals that both GMPO and GRPO are weighted forms of the policy gradient while the former enjoys more stable weights, which consequently benefits policy optimization and performance. Experiments on multiple mathematical reasoning benchmarks show that GMPO-7B improves the average Pass@1 of GRPO by up to 4.1%, outperforming many state-of-the-art approaches. Code is available at https://github.com/callsys/GMPO.

Key facts. Cited by 107 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 42 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

37 of 42 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  3. 2025BNPO: Beta Normalization Policy Optimization3 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025On-Policy RL with Optimal Reward Baseline29 May
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025ARM: Adaptive Reasoning Model26 May
  8. 2025Learning to Reason without External Rewards26 May
  9. 2025Stable Reinforcement Learning for Efficient Reasoning23 May
  10. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin20 May
  11. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  12. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
14 more
  1. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  4. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  5. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Process Reinforcement through Implicit Rewards3 Feb
  13. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  14. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2025Qwen2.5-VL Technical Reportcited by 5,499
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,369
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,111
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,077
  8. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 378
  9. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  10. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  11. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 31; 2026: 76.

107 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 200
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 177
  3. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  4. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 74
  5. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 68
  6. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  7. 2026Your Group-Relative Advantage Is Biasedcited by 32
  8. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
  9. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 26
  10. 2026On the Position Bias of On-Policy Distillationcited by 26
  11. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  12. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
38 more
  1. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulationcited by 19
  2. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 17
  3. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  4. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generationcited by 15
  5. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 14
  6. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 14
  7. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 14
  8. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 12
  9. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11
  10. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  11. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  12. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  13. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferencescited by 10
  14. 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 10
  15. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 10
  16. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 10
  17. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPOcited by 9
  18. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  19. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  20. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  21. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  22. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  23. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  24. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  25. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  26. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Expertscited by 6
  27. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
  28. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  29. 2025SSPO: Subsentence-level Policy Optimizationcited by 5
  30. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  31. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 5
  32. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  33. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 5
  34. 2025Thinking with Images via Self-Calling Agentcited by 4
  35. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4
  36. 2025Poivre: Self-Refining Visual Pointing with Reinforcement Learningcited by 3
  37. 2025Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learningcited by 3
  38. 2026ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Drivingcited by 3

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  2. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  3. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
  4. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
  5. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  6. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
  7. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  8. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning