Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement Learning with Verifiable Rewards (RLVR) is an essential paradigm that enhances the reasoning capabilities of Large Language Models (LLMs). However, existing methods typically rely on static policy optimization schemes that misalign with the model's evolving reasoning capabilities. To address this issue, we propose Adaptive Power-Mean Policy Optimization (APMPO), which comprises two main innovations: Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC). Specifically, PMPO introduces a generalized power-mean objective. This enables the model to adaptively transition from the signal-amplifying behavior of the arithmetic mean to the consistency-enforcing behavior of the geometric mean. FAC adaptively adjusts clipping bounds based on real-time reward statistics to overcome the limitations of static mechanisms. Capitalizing on these innovations, APMPO improves learning dynamics and reasoning performance. Extensive experiments on nine datasets across three reasoning tasks showcase the superiority of APMPO over state-of-the-art RLVR-based baselines. For instance, APMPO boosts the average Pass@1 score on mathematical reasoning benchmarks by 3.0 points compared to GRPO when using Qwen2.5-3B-Instruct.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 30 references that are in the map. First posted on arXiv 11 Apr 2026.
Builds on
18 of 30 references.
Closest ancestors · 8
Same direction, newest first
- 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct
- 2025Geometric-Mean Policy Optimization28 Jul
- 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
- 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning23 May
- 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
Other references · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024GPT-4o System Cardcited by 4,467
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
- 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
- 2025Process-Supervised Reinforcement Learning for Code Generationcited by 21
- 2025GeometryZero: Advancing Geometry Solving via Group Contrastive Policy Optimizationcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
- 2026What are Key Factors for Updates in RL for LLM Reasoning?
- 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
- 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
- 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
- 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
- 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
- 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning