mlsanityData to 7 Oct 2026

EffiBench: Benchmarking the Efficiency of Automatically Generated Code

arXiv 2402.02037 ↗First posted 3 Feb 2024cs.SE cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

Code generation models have increasingly become integral to aiding software development. Although current research has thoroughly examined the correctness of the code produced by code generation models, a vital aspect that plays a pivotal role in green computing and sustainability efforts has often been neglected. This paper presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems to assess the efficiency of code generated by code generation models. EffiBench contains a diverse set of LeetCode coding problems. Each problem is paired with an executable human-written canonical solution, which obtains the SOTA efficiency on the LeetCode solution leaderboard. With EffiBench, we empirically examine the ability of 42 large language models (35 open-source and 7 closed-source) to generate efficient code. Our evaluation results demonstrate that the efficiency of the code generated by LLMs is generally worse than the efficiency of human-written canonical solutions. For example, GPT-4 generated code has an average 3.12 times execution time that of the human-written canonical solutions. In the most extreme cases, the execution time and total memory usage of GPT-4 generated code are 13.89 and 43.92 times that of the canonical solutions. The source code of EffiBench is released on https://github.com/huangd1999/EffiBench. We also provide the LeaderBoard at https://huggingface.co/spaces/EffiBench/effibench-leaderboard.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 45 of its 88 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

45 of 88 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
33 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  2. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  5. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  6. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  10. 2020Unsupervised Translation of Programming Languagescited by 262
  11. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  12. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  13. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  14. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  15. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  16. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  17. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  18. 2023Learning Performance-Improving Code Editscited by 103
  19. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  20. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  21. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  22. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  23. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  24. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  25. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  26. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  27. 2023Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTcited by 36
  28. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  29. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  30. 2022AixBench: A Code Generation Benchmark Datasetcited by 22
  31. 2021Type4Py: Practical Deep Similarity Learning-Based Type Inference for Pythoncited by 19
  32. 2023TypeT5: Seq2seq Type Inference using Static Analysiscited by 12
  33. 2023Source Code Data Augmentation for Deep Learning: A Surveycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 20; 2026: 30.

62 papers.

Sort:

By area:

  1. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  2. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  3. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  4. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  5. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 39
  6. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  7. 2025Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarkscited by 31
  8. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  9. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  11. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 25
  12. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
38 more
  1. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  2. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  3. 2024Generative Monoculture in Large Language Modelscited by 21
  4. 2025CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generationcited by 16
  5. 2024EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuningcited by 15
  6. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  7. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  8. 2025MaintainCoder: Maintainable Code Generation Under Dynamic Requirementscited by 13
  9. 2025LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctnesscited by 10
  10. 2025Regression Language Models for Codecited by 10
  11. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 9
  12. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 9
  13. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
  14. 2025Generating Energy-Efficient Code via Large-Language Models -- Where are we now?cited by 5
  15. 2025Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesiscited by 4
  16. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  17. 2025Evaluating the Energy-Efficiency of the Code Generated by LLMscited by 3
  18. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  19. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  20. 2026PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimizationcited by 2
  21. 2026Evaluating LLMs on Real-World Software Performance Optimizationcited by 2
  22. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  23. 2025Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programmingcited by 1
  24. 2026AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptationcited by 1
  25. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  26. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  27. 2025SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMscited by 0
  28. 2025Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Softwarecited by 0
  29. 2026EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systemscited by 0
  30. 2026MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?cited by 0
  31. 2026EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedbackcited by 0
  32. 2026Task Abstention for Large Language Models in Code Generationcited by 0
  33. 2026AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Codecited by 0
  34. 2026CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Modelscited by 0
  35. 2026CodegenBench: Can LLMs Write Efficient Code Across Architectures?cited by 0
  36. 2026Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report)cited by 0
  37. 2026Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translationcited by 0
  38. 2026Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learningcited by 0

50 most cited of 62 All 62 (JSON)