mlsanityData to 7 Oct 2026

Evaluating Language Models for Efficient Code Generation

arXiv 2408.06450 ↗First posted 12 Aug 2024cs.SE cs.CL cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Aug 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 36; 2026: 26.

69 papers.

Sort:

By area:

  1. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  2. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  3. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  4. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  5. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  6. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 75
  7. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  8. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 64
  9. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  10. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 55
  11. 2025LongCat-Flash Technical Reportcited by 54
  12. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
38 more
  1. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
  2. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  3. 2025LongCat-Flash-Omni Technical Reportcited by 23
  4. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  5. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  6. 2025GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generationcited by 17
  7. 2025LIMI: Less is More for Agencycited by 17
  8. 2024EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuningcited by 15
  9. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  10. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  11. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  12. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  13. 2025Rethinking Verification for LLM Code Generation: From Generation to Testingcited by 12
  14. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  15. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  16. 2025LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctnesscited by 10
  17. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 9
  18. 2026EffiSkill: Agent Skill Based Automated Code Efficiency Optimizationcited by 9
  19. 2025Towards Better Correctness and Efficiency in Code Generationcited by 8
  20. 2026Hierarchical Sparse Attention Done Right: Toward Infinite Context Modelingcited by 8
  21. 2025Validating Network Protocol Parsers with Traceable RFC Document Interpretationcited by 7
  22. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 7
  23. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  24. 2025Large Language Models for Validating Network Protocol Parserscited by 5
  25. 2025SuperCoder: Assembly Program Superoptimization with Large Language Modelscited by 5
  26. 2025dots.llm1 Technical Reportcited by 5
  27. 2025The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMscited by 5
  28. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  29. 2026SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agentscited by 4
  30. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  31. 2026DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Buildercited by 3
  32. 2026ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenascited by 3
  33. 2026CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphscited by 3
  34. 2026Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decisioncited by 2
  35. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  36. 2025Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transformcited by 1
  37. 2025Program Semantic Inequivalence Game with Large Language Modelscited by 1
  38. 2026TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Modelscited by 1

50 most cited of 69 All 69 (JSON)