mlsanityData to 14 Oct 2026

GAPO: Robust Advantage Estimation for Real-World Code LLMs

arXiv 2510.21830 ↗First posted 22 Oct 2025cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, where group-relative methods, such as GRPO, are popular due to their critic-free and normalized advantage estimation. However, in real-world code-editing scenarios, reward distributions are often skewed with unpredictable noise, leading to distorted advantage computation and increased rollout outliers. To address this issue, we propose Group Adaptive Policy Optimization (GAPO), which adaptively finds an interval with the highest SNR (Signal to Noise Ratio) per prompt and uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation to reduce noise further. This adaptive Q robustly handles rollout noise while remaining plug-and-play and efficient. We evaluate GAPO on nine instruction-tuned LLMs (3B-14B) using a collected large dataset of 51,844 real-world, history-aware code-editing tasks spanning 10 programming languages. GAPO yields up to 4.35 in-domain (ID) and 5.30 out-of-domain (OOD) exact-match improvements over GRPO and its variant DAPO, while achieving lower clipping ratios and higher GPU throughput. Code: https://github.com/TsingZ0/verl-GAPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 56 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

35 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO29 Sep
  3. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning26 Sep
  4. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  5. 2025Geometric-Mean Policy Optimization28 Jul
  6. 2025ReDit: Reward Dithering for Improved LLM Policy Optimization23 Jun
  7. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  8. 2025On-Policy RL with Optimal Reward Baseline29 May
  9. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  10. 2025From Reasoning to Code: GRPO Optimization for Underrepresented Languages20 May
  11. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning12 May
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
1 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  11. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  12. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
10 more
  1. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  2. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 83
  3. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  4. 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 40
  5. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 22
  6. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
  7. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  8. 2025Enhancing Project-Specific Code Completion by Inferring Internal API Informationcited by 5
  9. 2025EditLord: Learning Code Transformation Rules for Code Editingcited by 4
  10. 2022Smallest covering regions and highest density regions for discrete distributionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
  2. 2026Group Adaptive Clipping Policy Optimization
  3. 2026Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
  4. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
  5. 2026Noisy Test-Time Reinforcement Learning for Code LLMs
  6. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization
  7. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
  8. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization