# CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models

arXiv: https://arxiv.org/abs/2302.00288 · HTML: https://mlsanity.com/paper/2302.00288/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2023.

## Built on by (most cited first, 28 of 28)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback](https://mlsanity.com/paper/2307.14936/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation](https://mlsanity.com/paper/2503.07010/) (2025)
- [CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks](https://mlsanity.com/paper/2404.00566/) (2024)
- [Guiding Language Models of Code with Global Context using Monitors](https://mlsanity.com/paper/2306.10763/) (2023)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions](https://mlsanity.com/paper/2304.03816/) (2023)
- [Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs](https://mlsanity.com/paper/2406.18294/) (2024)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models](https://mlsanity.com/paper/2406.01359/) (2024)
- [PatentGPT: A Large Language Model for Intellectual Property](https://mlsanity.com/paper/2404.18255/) (2024)
- [CodeShell Technical Report](https://mlsanity.com/paper/2403.15747/) (2024)
- [DCA-Bench: A Benchmark for Dataset Curation Agents](https://mlsanity.com/paper/2406.07275/) (2024)
- [Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks](https://mlsanity.com/paper/2406.15325/) (2024)
- [FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration](https://mlsanity.com/paper/2510.04852/) (2025)
- [Can Language Models Pretend Solvers? Logic Code Simulation with LLMs](https://mlsanity.com/paper/2403.16097/) (2024)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?](https://mlsanity.com/paper/2309.07804/) (2023)
- [Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4](https://mlsanity.com/paper/2402.12782/) (2024)
- [TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation](https://mlsanity.com/paper/2509.24148/) (2025)
- [GUI Test Migration via Abstraction and Concretization](https://mlsanity.com/paper/2409.05028/) (2024)
- [CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation](https://mlsanity.com/paper/2504.20673/) (2025)
- [Multi-task Code LLMs: Data Mix or Model Merge?](https://mlsanity.com/paper/2601.21115/) (2026)
- [IntentTester: Intent-Driven Multi-agent Framework for Cross-Library Test Migration](https://mlsanity.com/paper/2606.25588/) (2026)
- [An Empirical Study for Android-to-OpenHarmony GUI Test Migration](https://mlsanity.com/paper/2607.11245/) (2026)

## Abstract (arXiv)

Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.