mlsanityData to 14 Oct 2026

GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning

arXiv 2508.17850 ↗First posted 25 Aug 2025cs.LG cs.AIcited by 4

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees. Experiments show GEPO achieves superior stability - only a 3% performance drop from online to 1800s latency-and reduces the best-to-last gap by 85% versus GSPO (1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 32 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

26 of 32 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
14 more
  1. 2025Group Sequence Policy Optimizationcited by 835
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  6. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  7. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  8. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  9. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  10. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  11. 2025BNPO: Beta Normalization Policy Optimizationcited by 28
  12. 2025Stable Reinforcement Learning for Efficient Reasoningcited by 27
  13. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learningcited by 25
  14. 2023Reward Collapse in Aligning Large Language Modelscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  3. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  4. 2026VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
  2. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization
  3. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing
  4. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
  5. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPO
  6. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  7. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  8. 2026Stabilizing Reinforcement Learning for Diffusion Language Models