# GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning

arXiv: https://arxiv.org/abs/2507.10628 · HTML: https://mlsanity.com/paper/2507.10628/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-play RL for reasoning](https://mlsanity.com/synth/self-play-rl-for-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)

## Built on by (most cited first, 30 of 34)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation](https://mlsanity.com/paper/2601.20614/) (2026)
- [Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts](https://mlsanity.com/paper/2510.09388/) (2025)
- [Automated Optimization Modeling via a Localizable Error-Driven Perspective](https://mlsanity.com/paper/2602.11164/) (2026)
- [EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.03108/) (2026)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs](https://mlsanity.com/paper/2512.00908/) (2025)
- [No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning](https://mlsanity.com/paper/2601.06794/) (2026)
- [Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning](https://mlsanity.com/paper/2602.20722/) (2026)
- [TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents](https://mlsanity.com/paper/2602.21230/) (2026)
- [KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance](https://mlsanity.com/paper/2604.12627/) (2026)
- [Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs](https://mlsanity.com/paper/2510.04140/) (2025)
- [More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration](https://mlsanity.com/paper/2510.02227/) (2025)
- [Verifying Meta-Awareness via Predictive Rewards in Reasoning Models](https://mlsanity.com/paper/2510.03259/) (2025)
- [Toward Honest Language Models for Deductive Reasoning](https://mlsanity.com/paper/2511.09222/) (2025)
- [Agentic Policy Optimization via Instruction-Policy Co-Evolution](https://mlsanity.com/paper/2512.01945/) (2025)
- [Context Learning for Multi-Agent Discussion](https://mlsanity.com/paper/2602.02350/) (2026)
- [HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation](https://mlsanity.com/paper/2603.10359/) (2026)
- [P^2O: Joint Policy and Prompt Optimization](https://mlsanity.com/paper/2603.21877/) (2026)
- [Efficient Reasoning via Reward Model](https://mlsanity.com/paper/2511.09158/) (2025)
- [P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering](https://mlsanity.com/paper/2601.20649/) (2026)
- [Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance](https://mlsanity.com/paper/2604.23318/) (2026)
- [Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective](https://mlsanity.com/paper/2605.12969/) (2026)
- [Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models](https://mlsanity.com/paper/2601.04731/) (2026)
- [Context Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2603.18953/) (2026)
- [PAL: Personal Adaptive Learner](https://mlsanity.com/paper/2604.13017/) (2026)
- [The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms](https://mlsanity.com/paper/2606.25450/) (2026)
- [Offline-Online Curriculum RL for Multimodal Reasoning](https://mlsanity.com/paper/2607.23700/) (2026)
- [AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning](https://mlsanity.com/paper/2607.24833/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for facilitating the self-improvement of large language models (LLMs), particularly in the domain of complex reasoning tasks. However, prevailing on-policy RL methods often contend with significant training instability and inefficiency. This is primarily due to a capacity-difficulty mismatch, where the complexity of training data frequently outpaces the model's current capabilities, leading to critically sparse reward signals and stalled learning progress. This challenge is particularly acute for smaller, more resource-efficient LLMs. To overcome this, we introduce the Guided Hybrid Policy Optimization (GHPO), a novel difficulty-aware reinforcement learning framework. GHPO dynamically calibrates task difficulty by employing adaptive prompt refinement to provide targeted guidance. This unique approach adaptively balances direct imitation learning for problems currently beyond the model's reach with exploration-based reinforcement learning for more manageable tasks, effectively creating a smooth and optimized learning curriculum. Extensive experiments demonstrate that GHPO achieves an average performance gain of approximately 5% across six challenging mathematics benchmarks, consistently outperforming strong on-policy reinforcement learning and curriculum learning baselines. Further analysis confirms that our framework significantly enhances both training stability and final reasoning performance, thus offering a scalable and efficient solution for developing powerful and robust reasoning models.