mlsanityData to 7 Oct 2026

EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

arXiv 2505.13004 ↗First posted 19 May 2025cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around 62% of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

49 of 72 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
37 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  2. 2023StarCoder: may the source be with you!cited by 790
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  7. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  10. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  11. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  12. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  13. 2025Phi-4-reasoning Technical Reportcited by 146
  14. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  15. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  16. 2023Learning Performance-Improving Code Editscited by 103
  17. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  18. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  19. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  20. 2024Evaluating Language Models for Efficient Code Generationcited by 69
  21. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  22. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  23. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  24. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  25. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 39
  26. 2023Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTcited by 36
  27. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  28. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  29. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 25
  30. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  31. 2021Type4Py: Practical Deep Similarity Learning-Based Type Inference for Pythoncited by 19
  32. 2024On Evaluating the Efficiency of Source Code Generated by LLMscited by 19
  33. 2024EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuningcited by 15
  34. 2023TypeT5: Seq2seq Type Inference using Static Analysiscited by 12
  35. 2024ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Modelscited by 10
  36. 2025CodeArena: A Collective Evaluation Platform for LLM Code Generationcited by 10
  37. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 16.

24 papers.

Sort:

By area:

  1. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  2. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  3. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  5. 2026Controlled Self-Evolution for Algorithmic Code Optimizationcited by 11
  6. 2026EffiSkill: Agent Skill Based Automated Code Efficiency Optimizationcited by 9
  7. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  8. 2026Agent Skill Evaluation and Evolution: Frameworks and Benchmarkscited by 6
  9. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  10. 2025Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesiscited by 4
  11. 2026CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generationcited by 2
  12. 2026PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimizationcited by 2
12 more
  1. 2026Evaluating LLMs on Real-World Software Performance Optimizationcited by 2
  2. 2025Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacycited by 1
  3. 2026Comparing Developer and LLM Biases in Code Evaluationcited by 1
  4. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  5. 2026The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languagescited by 1
  6. 2025SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMscited by 0
  7. 2025Rethinking Technology Stack Selection with AI Coding Proficiencycited by 0
  8. 2026EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systemscited by 0
  9. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  10. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0
  11. 2026RLPF: Reinforcement Learning from Performance Feedback for Code Generationcited by 0
  12. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Models
  2. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  3. 2026CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
  4. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  5. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks
  6. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  7. 2025OckBench: Measuring the Efficiency of LLM Reasoning
  8. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories