mlsanityData to 7 Oct 2026

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

arXiv 2405.19856 ↗First posted 30 May 2024cs.CL cs.SEcited by 71

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

26 of 32 references.

References · 26

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
14 more
  1. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  2. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  3. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  4. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  5. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  6. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  7. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  8. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  9. 2018Mapping Language to Code in Programmatic Contextcited by 104
  10. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  11. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  12. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40
  13. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  14. 2022Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 28; 2026: 30.

71 papers.

Sort:

By area:

  1. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  2. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  3. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 57
  4. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  5. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  6. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  7. 2025MultiKernelBench: A Multi-Platform Benchmark for Kernel Generationcited by 35
  8. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 33
  9. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generationcited by 33
  10. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  11. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 24
  12. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
38 more
  1. 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
  2. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 20
  3. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  4. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  5. 2026In Line with Context: Repository-Level Code Generation via Context Inliningcited by 18
  6. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 17
  7. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  8. 2025GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generationcited by 17
  9. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  10. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  11. 2026OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Codingcited by 15
  12. 2025CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generationcited by 14
  13. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 14
  14. 2025SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Developmentcited by 13
  15. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  16. 2026A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compressioncited by 12
  17. 2026Structurally Aligned Subtask-Level Memory for Software Engineering Agentscited by 11
  18. 2024JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Modelscited by 10
  19. 2024RepoMasterEval: Evaluating Code Completion via Real-World Repositoriescited by 10
  20. 2026ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Developmentcited by 9
  21. 2025FeatBench: Towards More Realistic Evaluation of Feature-level Code Generationcited by 6
  22. 2025Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modelingcited by 6
  23. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactionscited by 6
  24. 2025YABLoCo: Yet Another Benchmark for Long Context Code Generationcited by 5
  25. 2025FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migrationcited by 5
  26. 2026JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Enginescited by 5
  27. 2025A Study on Thinking Patterns of Large Reasoning Models in Code Generationcited by 4
  28. 2026Multi-agent Collaboration with State Managementcited by 4
  29. 2026When Parallelism Pays Off: Cohesion-Aware Task Partitioning for Multi-Agent Codingcited by 4
  30. 2024Towards Exception Safety Code Generation with Intermediate Representation Agents Frameworkcited by 3
  31. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  32. 2025Evaluating the Generalizability of LLMs in Automated Program Repaircited by 3
  33. 2025Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Constructioncited by 3
  34. 2024Seeker: Towards Exception Safety Code Generation with Intermediate Language Agents Frameworkcited by 2
  35. 2025FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verificationcited by 2
  36. 2025Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEvalcited by 2
  37. 2025TENET: One Step Toward Test-Driven Development for Repository-Level Code Generationcited by 2
  38. 2026ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generationcited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
  2. 2025MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
  3. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  4. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  5. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  6. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  7. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  8. 2024A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models