# Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization

arXiv: https://arxiv.org/abs/2502.01652 · HTML: https://mlsanity.com/paper/2502.01652/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2025.

## Built on by (most cited first, 15 of 15)

- [EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.04623/) (2025)
- [CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment](https://mlsanity.com/paper/2508.02298/) (2025)
- [Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach](https://mlsanity.com/paper/2503.21819/) (2025)
- [Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models](https://mlsanity.com/paper/2505.17697/) (2025)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation](https://mlsanity.com/paper/2605.21125/) (2026)
- [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025)
- [DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models](https://mlsanity.com/paper/2507.09955/) (2025)
- [Group Policy Gradient](https://mlsanity.com/paper/2510.03679/) (2025)
- [Policy Optimized Text-to-Image Pipeline Design](https://mlsanity.com/paper/2505.21478/) (2025)
- [Interaction as Intelligence Part II: Asynchronous Human-Agent Rollout for Long-Horizon Task Training](https://mlsanity.com/paper/2510.27630/) (2025)
- [ScRPO: From Errors to Insights](https://mlsanity.com/paper/2511.06065/) (2025)
- [Consensus Group Relative Policy Optimization for Text Generation](https://mlsanity.com/paper/2602.03102/) (2026)
- [Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery](https://mlsanity.com/paper/2609.19164/) (2026)
- [Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data](https://mlsanity.com/paper/2511.06023/) (2025)

## Abstract (arXiv)

Hybrid Group Relative Policy Optimization (Hybrid GRPO) is a reinforcement learning framework that extends Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) by incorporating empirical multi-sample action evaluation while preserving the stability of value function-based learning. Unlike DeepSeek GRPO, which eliminates the value function in favor of purely empirical reward estimation, Hybrid GRPO introduces a structured advantage computation method that balances empirical action sampling with bootstrapped value estimation. This approach enhances sample efficiency, improves learning stability, and mitigates variance amplification observed in purely empirical methods. A detailed mathematical comparison between PPO, DeepSeek GRPO, and Hybrid GRPO is presented, highlighting key differences in advantage estimation and policy updates. Experimental validation in a controlled reinforcement learning environment demonstrates that Hybrid GRPO achieves superior convergence speed, more stable policy updates, and improved sample efficiency compared to existing methods. Several extensions to Hybrid GRPO are explored, including entropy-regularized sampling, hierarchical multi-step sub-sampling, adaptive reward normalization, and value-based action selection. Beyond reinforcement learning in simulated environments, Hybrid GRPO provides a scalable framework for bridging the gap between large language models (LLMs) and real-world agent-based decision-making. By integrating structured empirical sampling with reinforcement learning stability mechanisms, Hybrid GRPO has potential applications in autonomous robotics, financial modeling, and AI-driven control systems. These findings suggest that Hybrid GRPO serves as a robust and adaptable reinforcement learning methodology, paving the way for further advancements in policy optimization.