OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research
Not on the map. Why?
Abstract (arXiv)
Existing class-level code generation datasets are either synthetic (ClassEval: 100 classes) or insufficient in scale for modern training needs (RealClassEval: 400 classes), hindering robust evaluation and empirical analysis. We present OpenClassGen, a large-scale corpus of 324,843 Python classes extracted from 2,970 engineered open-source projects. Each entry pairs a human-written class with its corresponding skeleton, which comprises class and method signatures with associated docstrings, and is enriched with 27 static code metrics covering complexity, coupling, cohesion, and inheritance properties. Unlike prior benchmarks that require repository-level context resolution, OpenClassGen provides self-contained class skeletons that serve as complete generation specifications. We demonstrate the corpus's utility by evaluating three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a curated, executable subset of 300 classes, enriched with test suites achieving 58% branch coverage. Results show strong semantic similarity (CodeBERTScore-F3: 0.89) but moderate functional correctness (pass rate: 0.33), with substantial variance across models. This variance, along with diverse class characteristics, confirms that OpenClassGen enables meaningful differentiation of LLM capabilities. The dataset supports diverse use cases, including fine-tuning, retrieval-augmented generation, difficulty modelling, and failure mode analysis. The complete dataset and curation scripts are publicly available at https://zenodo.org/records/18409150.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 12 of its 39 references that are in the map. First posted on arXiv 22 Apr 2025.
Builds on
12 of 39 references.
References · 12
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
- 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
- 2018Mapping Language to Code in Programmatic Contextcited by 104
- 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
- 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
- 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
- 2024HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generationcited by 26
- 2024Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repositorycited by 14
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.
- 2026ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generationcited by 2
- 2025Lifecycle-Aware code generation: Leveraging Software Engineering Phases in LLMscited by 0
- 2026TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA Specification Generationcited by 0
- 2026TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA Specification Generation26 Jul 2026
- 2026ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation29 Apr 2026
- 2025Lifecycle-Aware code generation: Leveraging Software Engineering Phases in LLMs28 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
- 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
- 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
- 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models
- 2025MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
- 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
- 2026Evaluating LLMs Code Reasoning Under Real-World Context
- 2026Exploring different approaches to customize language models for domain-specific text-to-code generation