# EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code

arXiv: https://arxiv.org/abs/2505.13004 · HTML: https://mlsanity.com/paper/2505.13004/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 72 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation](https://mlsanity.com/paper/2206.06888/) (2022)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)

## Built on by (most cited first, 27 of 27)

- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization](https://mlsanity.com/paper/2505.23387/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Controlled Self-Evolution for Algorithmic Code Optimization](https://mlsanity.com/paper/2601.07348/) (2026)
- [EffiSkill: Agent Skill Based Automated Code Efficiency Optimization](https://mlsanity.com/paper/2603.27850/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation](https://mlsanity.com/paper/2602.03045/) (2026)
- [CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models](https://mlsanity.com/paper/2602.17684/) (2026)
- [Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis](https://mlsanity.com/paper/2510.26423/) (2025)
- [CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation](https://mlsanity.com/paper/2604.12268/) (2026)
- [PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization](https://mlsanity.com/paper/2605.15222/) (2026)
- [Evaluating LLMs on Real-World Software Performance Optimization](https://mlsanity.com/paper/2606.25530/) (2026)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [Comparing Developer and LLM Biases in Code Evaluation](https://mlsanity.com/paper/2603.24586/) (2026)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages](https://mlsanity.com/paper/2607.22807/) (2026)
- [SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements](https://mlsanity.com/paper/2607.27409/) (2026)
- [SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs](https://mlsanity.com/paper/2506.11081/) (2025)
- [Rethinking Technology Stack Selection with AI Coding Proficiency](https://mlsanity.com/paper/2509.11132/) (2025)
- [EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems](https://mlsanity.com/paper/2602.10171/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs](https://mlsanity.com/paper/2607.12273/) (2026)
- [Reinforcement Learning for Code Optimization](https://mlsanity.com/paper/2607.25970/) (2026)
- [RLPF: Reinforcement Learning from Performance Feedback for Code Generation](https://mlsanity.com/paper/2607.27271/) (2026)
- [Collective Bias Mitigation via Model Routing and Collaboration](https://mlsanity.com/paper/2610.03240/) (2026)
- [Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution](https://mlsanity.com/paper/2610.13563/) (2026)

## Abstract (arXiv)

Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X, the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around 62% of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1's Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x.