# ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?

arXiv: https://arxiv.org/abs/2407.14044 · HTML: https://mlsanity.com/paper/2407.14044/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)
- [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024)
- [Large Language Models for Compiler Optimization](https://mlsanity.com/paper/2309.07062/) (2023)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [On Evaluating the Efficiency of Source Code Generated by LLMs](https://mlsanity.com/paper/2404.06041/) (2024)
- [Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency](https://mlsanity.com/paper/2406.12502/) (2024)
- [Learning based Methods for Code Runtime Complexity Prediction](https://mlsanity.com/paper/1911.01155/) (2019)
- [Learning to Superoptimize Real-world Programs](https://mlsanity.com/paper/2109.13498/) (2021)
- [A Problem-Oriented Perspective and Anchor Verification for Code Optimization](https://mlsanity.com/paper/2406.11935/) (2024)
- [TASTY: A Transformer based Approach to Space and Time complexity](https://mlsanity.com/paper/2305.05379/) (2023)

## Built on by (most cited first, 27 of 27)

- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code](https://mlsanity.com/paper/2505.13004/) (2025)
- [AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?](https://mlsanity.com/paper/2507.15887/) (2025)
- [GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents](https://mlsanity.com/paper/2505.23671/) (2025)
- [SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?](https://mlsanity.com/paper/2511.06090/) (2025)
- [Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization](https://mlsanity.com/paper/2505.23387/) (2025)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctness](https://mlsanity.com/paper/2502.18489/) (2025)
- [RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation](https://mlsanity.com/paper/2503.22851/) (2025)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [EditLord: Learning Code Transformation Rules for Code Editing](https://mlsanity.com/paper/2504.15284/) (2025)
- [Refactoring Codebases through Library Design](https://mlsanity.com/paper/2506.11058/) (2025)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [PerfCoder: Large Language Models for Interpretable Code Performance Optimization](https://mlsanity.com/paper/2512.14018/) (2025)
- [AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming](https://mlsanity.com/paper/2507.15378/) (2025)
- [MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?](https://mlsanity.com/paper/2603.11935/) (2026)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Code](https://mlsanity.com/paper/2604.02352/) (2026)
- [EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback](https://mlsanity.com/paper/2604.05137/) (2026)
- [AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code](https://mlsanity.com/paper/2605.17978/) (2026)
- [Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation](https://mlsanity.com/paper/2606.17683/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation](https://mlsanity.com/paper/2607.20908/) (2026)
- [Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution](https://mlsanity.com/paper/2610.13563/) (2026)
- [HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation](https://mlsanity.com/paper/2610.19184/) (2026)

## Abstract (arXiv)

Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.