# ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models

arXiv: https://arxiv.org/abs/2412.17264 · HTML: https://mlsanity.com/paper/2412.17264/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2024.

## Built on by (most cited first, 10 of 10)

- [EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code](https://mlsanity.com/paper/2505.13004/) (2025)
- [RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System](https://mlsanity.com/paper/2602.02488/) (2026)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Towards Better Correctness and Efficiency in Code Generation](https://mlsanity.com/paper/2508.20124/) (2025)
- [Improving LLM-Generated Code Quality with GRPO](https://mlsanity.com/paper/2506.02211/) (2025)
- [LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages](https://mlsanity.com/paper/2503.19217/) (2025)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning](https://mlsanity.com/paper/2503.14838/) (2025)
- [Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models](https://mlsanity.com/paper/2507.17389/) (2025)
- [VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning](https://mlsanity.com/paper/2508.18462/) (2025)

## Abstract (arXiv)

CodeLLMs have demonstrated remarkable advancements in software engineering tasks. However, while these models can generate functionally correct code, they often produce code that is inefficient in terms of runtime. This inefficiency is particularly problematic in resource-constrained environments, impacting software performance and sustainability. Existing approaches for optimizing code efficiency for CodeLLMs like SOAP and PIE exhibit certain limitations. SOAP requires a compatible execution environment and predefined test cases for iterative code modification, while PIE focuses on instruction tuning, improving efficiency but compromising correctness. These shortcomings highlight the need for a fine-tuning framework that optimizes both efficiency and correctness without relying on predefined test cases or specific execution environments. To bridge this gap, we introduce ACECode, a reinforcement learning-based fine-tuning framework that aligns CodeLLMs with dual objectives of efficiency and correctness. ACECode combines three key steps: (1) generating code with an actor CodeLLM, (2) calculating a training-free reward signal derived from code execution feedback for each generated code, and (3) optimizing the CodeLLM via Proximal Policy Optimization (PPO) algorithm. This reward signal enables joint assessment of efficiency and correctness without manual labeling. We evaluate ACECode by fine-tuning four SOTA (state-of-the-art) CodeLLMs and comparing their code with three baselines: original, instruction-tuned, and PIE-tuned CodeLLMs. Extensive experiment results suggest that tool significantly improves the efficiency and correctness of generated code against all baselines for all CodeLLMs. Specifically, CodeLLMs fine-tuned with ACECode improve pass@1 by 1.84% to 14.51% and reduce runtime in 65% to 72% of cases compared to original CodeLLMs.