mlsanityData to 7 Oct 2026

OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research

arXiv 2504.15564 ↗First posted 22 Apr 2025cs.SE cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Existing class-level code generation datasets are either synthetic (ClassEval: 100 classes) or insufficient in scale for modern training needs (RealClassEval: 400 classes), hindering robust evaluation and empirical analysis. We present OpenClassGen, a large-scale corpus of 324,843 Python classes extracted from 2,970 engineered open-source projects. Each entry pairs a human-written class with its corresponding skeleton, which comprises class and method signatures with associated docstrings, and is enriched with 27 static code metrics covering complexity, coupling, cohesion, and inheritance properties. Unlike prior benchmarks that require repository-level context resolution, OpenClassGen provides self-contained class skeletons that serve as complete generation specifications. We demonstrate the corpus's utility by evaluating three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a curated, executable subset of 300 classes, enriched with test suites achieving 58% branch coverage. Results show strong semantic similarity (CodeBERTScore-F3: 0.89) but moderate functional correctness (pass rate: 0.33), with substantial variance across models. This variance, along with diverse class characteristics, confirms that OpenClassGen enables meaningful differentiation of LLM capabilities. The dataset supports diverse use cases, including fine-tuning, retrieval-augmented generation, difficulty modelling, and failure mode analysis. The complete dataset and curation scripts are publicly available at https://zenodo.org/records/18409150.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 12 of its 39 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

12 of 39 references.

References · 12

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  6. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  7. 2018Mapping Language to Code in Programmatic Contextcited by 104
  8. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  9. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  10. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  11. 2024HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generationcited by 26
  12. 2024Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repositorycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  2. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
  3. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
  4. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models
  5. 2025MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
  6. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  7. 2026Evaluating LLMs Code Reasoning Under Real-World Context
  8. 2026Exploring different approaches to customize language models for domain-specific text-to-code generation