mlsanityData to 7 Oct 2026

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

arXiv 2404.00599 ↗First posted 31 Mar 2024cs.CL cs.AI cs.SEcited by 71

Not on the map. Why?

Abstract (arXiv)

How to evaluate Large Language Models (LLMs) in code generation is an open question. Existing benchmarks demonstrate poor alignment with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. This paper proposes a new benchmark - EvoCodeBench to address the preceding problems, which has three primary advances. (1) EvoCodeBench aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) EvoCodeBench offers comprehensive annotations (e.g., requirements, reference code, and reference dependencies), and robust evaluation metrics (e.g., Pass@k and Recall@k). (3) EvoCodeBench is an evolving benchmark to avoid data leakage. We build an automatic pipeline to update EvoCodeBench from the latest repositories. We release the first version - EvoCodeBench-2403, containing 275 samples from 25 real-world repositories. Based on EvoCodeBench, we propose repository-level code generation and evaluate 10 popular LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, CodeLLaMa, Gemma, and Qwen 1.5). Our experiments reveal the coding abilities of these LLMs in real-world repositories. For example, the highest Pass@1 of gpt-4 only is 20.73% in our experiments. We also analyze failed cases and summarize the shortcomings of existing LLMs in EvoCodeBench. We release EvoCodeBench, all prompts, and LLMs' completions for further community analysis.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

29 of 35 references.

References · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
17 more
  1. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  2. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  3. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  4. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  5. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  6. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  7. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  8. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  9. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  10. 2018Mapping Language to Code in Programmatic Contextcited by 104
  11. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  12. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 71
  13. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  14. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40
  15. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  16. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  17. 2022Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 41; 2026: 17.

71 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  3. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  4. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  5. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  6. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 57
  7. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  8. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  9. 2025Repo2Run: Automated Building Executable Environment for Code Repository at Scalecited by 39
  10. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  11. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  12. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
38 more
  1. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  2. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 24
  3. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  4. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  5. 2026In Line with Context: Repository-Level Code Generation via Context Inliningcited by 18
  6. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 17
  7. 2025GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generationcited by 17
  8. 2025DependEval: Benchmarking LLMs for Repository Dependency Understandingcited by 16
  9. 2025CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generationcited by 16
  10. 2025Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verificationcited by 16
  11. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  12. 2026OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Codingcited by 15
  13. 2024Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMscited by 13
  14. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
  15. 2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 13
  16. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  17. 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoningcited by 12
  18. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  19. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  20. 2024RepoMasterEval: Evaluating Code Completion via Real-World Repositoriescited by 10
  21. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10
  22. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
  23. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  24. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
  25. 2025DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scalecited by 7
  26. 2025DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generationcited by 7
  27. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
  28. 2025FeatBench: Towards More Realistic Evaluation of Feature-level Code Generationcited by 6
  29. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactionscited by 6
  30. 2025YABLoCo: Yet Another Benchmark for Long Context Code Generationcited by 5
  31. 2025Impact-driven Context Filtering For Cross-file Code Completioncited by 5
  32. 2025Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigmcited by 5
  33. 2025FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migrationcited by 5
  34. 2026KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?cited by 5
  35. 2026JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Enginescited by 5
  36. 2024NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkerscited by 4
  37. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4
  38. 2025Knowledge Graph Based Repository-Level Code Generationcited by 4

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
  3. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  4. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  5. 2025MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
  6. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  7. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
  8. 2024EVOR: Evolving Retrieval for Code Generation