mlsanityData to 14 Oct 2026

ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation

arXiv 2308.01861 ↗First posted 3 Aug 2023cs.CL cs.AIcited by 168

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes. Our benchmark is available at https://github.com/FudanSELab/ClassEval.

Key facts. Cited by 168 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

36 of 51 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
24 more
  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2023StarCoder: may the source be with you!cited by 799
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  9. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  10. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  11. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  12. 2023SantaCoder: don't reach for the stars!cited by 170
  13. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  14. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  15. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  16. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 111
  17. 2018Mapping Language to Code in Programmatic Contextcited by 104
  18. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  19. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  20. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 48
  21. 2023Explainable Automated Debugging via Large Language Model-driven Scientific Debuggingcited by 40
  22. 2015Discrete Signal Processing on Graphs: Sampling Theorycited by 34
  23. 2023Can Large Language Models Write Good Property-Based Tests?cited by 31
  24. 2023CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Modelscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 65; 2025: 53; 2026: 39.

168 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2024A Survey on Large Language Models for Code Generationcited by 658
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  5. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  6. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  8. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  9. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 93
  10. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  11. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  12. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
38 more
  1. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  2. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  3. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
  4. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  5. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  6. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  7. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 58
  8. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
  9. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  10. 2024Commit0: Library Generation from Scratchcited by 52
  11. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 51
  12. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 46
  13. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 45
  14. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
  15. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  16. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  17. 2024Can Large Language Models Write Parallel Code?cited by 36
  18. 2024Automatic Programming: Large Language Models and Beyondcited by 36
  19. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 32
  20. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 31
  21. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  22. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 28
  23. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  24. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  25. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 25
  26. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  27. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  28. 2024PyBench: Evaluating LLM Agent on various real-world coding taskscited by 24
  29. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Taskscited by 24
  30. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  31. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  32. 20243D-PreMise: Can Large Language Models Generate 3D Shapes with Sharp Features and Parametric Control?cited by 21
  33. 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
  34. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  35. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 19
  36. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  37. 2025Unified Software Engineering Agent as AI Software Engineercited by 19
  38. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 18

50 most cited of 168 All 168 (JSON)