# Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

arXiv: https://arxiv.org/abs/2505.23387 · HTML: https://mlsanity.com/paper/2505.23387/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 52 of its 74 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)
- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 30 of 52)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://mlsanity.com/paper/2402.16906/) (2024)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)

## Built on by (most cited first, 18 of 18)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [Controlled Self-Evolution for Algorithmic Code Optimization](https://mlsanity.com/paper/2601.07348/) (2026)
- [Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model](https://mlsanity.com/paper/2602.07422/) (2026)
- [Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment](https://mlsanity.com/paper/2508.04865/) (2025)
- [MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimization](https://mlsanity.com/paper/2601.05475/) (2026)
- [Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL](https://mlsanity.com/paper/2605.28751/) (2026)
- [Automated Design Optimization via Strategic Search with Large Language Models](https://mlsanity.com/paper/2511.22651/) (2025)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution](https://mlsanity.com/paper/2603.02510/) (2026)
- [An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Code](https://mlsanity.com/paper/2604.02352/) (2026)
- [Paper Espresso: From Paper Overload to Research Insight](https://mlsanity.com/paper/2604.04562/) (2026)
- [AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code](https://mlsanity.com/paper/2605.17978/) (2026)
- [Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction](https://mlsanity.com/paper/2607.01764/) (2026)
- [Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning](https://mlsanity.com/paper/2607.04577/) (2026)
- [Reinforcement Learning for Code Optimization](https://mlsanity.com/paper/2607.25970/) (2026)
- [Collective Bias Mitigation via Model Routing and Collaboration](https://mlsanity.com/paper/2610.03240/) (2026)
- [Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution](https://mlsanity.com/paper/2610.13563/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) generate functionally correct solutions but often fall short in code efficiency, a critical bottleneck for real-world deployment. In this paper, we introduce a novel test-time iterative optimization framework to address this, employing a closed-loop system where LLMs iteratively refine code based on empirical performance feedback from an execution sandbox. We explore three training strategies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). Experiments on our Venus dataset and the APPS benchmark show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO, using reinforcement learning (RL) with execution feedback, continuously optimizes code performance, significantly boosting both pass@1 (from 47% to 62%) and the likelihood of outperforming human submissions in efficiency (from 31% to 45%). Our work demonstrates effective test-time code efficiency improvement and critically reveals the power of RL in teaching LLMs to truly self-improve code efficiency.