mlsanityData to 14 Oct 2026

DevEval: Evaluating Code Generation in Practical Software Projects

arXiv 2401.06401 ↗First posted 12 Jan 2024cs.SE cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

14 of 18 references.

References · 14

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  6. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  7. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  8. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  9. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  10. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
  11. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  12. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
2 more
  1. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  2. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 4.

9 papers.

  1. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  2. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  3. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 46
  4. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 45
  5. 2025RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testingcited by 13
  6. 2024AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instructcited by 8
  7. 2025Enhancing Project-Specific Code Completion by Inferring Internal API Informationcited by 5
  8. 2025From Context to Intent: Reasoning-Guided Function-Level Code Completioncited by 1
  9. 2025TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Frameworkcited by 1