# Improving LLM-Generated Code Quality with GRPO

arXiv: https://arxiv.org/abs/2506.02211 · HTML: https://mlsanity.com/paper/2506.02211/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [Process Supervision-Guided Policy Optimization for Code Generation](https://mlsanity.com/paper/2410.17621/) (2024)
- [Process-Supervised Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2502.01715/) (2025)
- [ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models](https://mlsanity.com/paper/2412.17264/) (2024)
- [Iterative Self-Training for Code Generation via Reinforced Re-Ranking](https://mlsanity.com/paper/2504.09643/) (2025)

## Built on by (most cited first, 8 of 8)

- [FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning](https://mlsanity.com/paper/2509.21792/) (2025)
- [EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2602.23802/) (2026)
- [CoRe-Code: Collaborative Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2605.24812/) (2026)
- [Find, Fix, Reason: Context Repair for Video Reasoning](https://mlsanity.com/paper/2604.16243/) (2026)
- [Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training](https://mlsanity.com/paper/2604.16723/) (2026)
- [Information-Consistent Language Model Recommendations through Group Relative Policy Optimization](https://mlsanity.com/paper/2512.12858/) (2025)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [LintEval: Benchmarking LLMs on Linter-Guided File Refactoring](https://mlsanity.com/paper/2610.20027/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are gaining widespread use for code generation. Recent training procedures use execution feedback as a reward signal, typically focusing on the functional correctness of the code, using unit test pass rate as a reward signal. However, this reward signal fails to capture notions of maintainability, quality and safety of the code produced. We address this under-explored area and develop a comprehensive library to quantify various aspects of code quality, and use it as a reward in GRPO. We find GRPO increases code quality according to this measure, which is confirmed by expert, blinded human annotators.