mlsanityData to 7 Oct 2026

ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?

arXiv 2407.14044 ↗First posted 19 Jul 2024cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

22 of 32 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  12. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
10 more
  1. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  2. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  3. 2023Large Language Models for Compiler Optimizationcited by 52
  4. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  5. 2024On Evaluating the Efficiency of Source Code Generated by LLMscited by 19
  6. 2024Code-Optimise: Self-Generated Preference Data for Correctness and Efficiencycited by 16
  7. 2019Learning based Methods for Code Runtime Complexity Predictioncited by 5
  8. 2021Learning to Superoptimize Real-world Programscited by 4
  9. 2023TASTY: A Transformer based Approach to Space and Time complexitycited by 3
  10. 2024A Problem-Oriented Perspective and Anchor Verification for Code Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  2. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  3. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  4. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
  5. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  6. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  7. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  8. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  9. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  10. 2025LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctnesscited by 10
  11. 2025RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generationcited by 5
  12. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
13 more
  1. 2025EditLord: Learning Code Transformation Rules for Code Editingcited by 4
  2. 2025Refactoring Codebases through Library Designcited by 4
  3. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  4. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  5. 2025AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programmingcited by 2
  6. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  7. 2026MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?cited by 0
  8. 2026An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Codecited by 0
  9. 2026EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedbackcited by 0
  10. 2026AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Codecited by 0
  11. 2026Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translationcited by 0
  12. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  13. 2026Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generationcited by 0