# AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

arXiv: https://arxiv.org/abs/2508.09101 · HTML: https://mlsanity.com/paper/2508.09101/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2025.

## Built on by (most cited first, 26 of 26)

- [NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents](https://mlsanity.com/paper/2512.12730/) (2025)
- [SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents](https://mlsanity.com/paper/2604.17308/) (2026)
- [Aegis: Automated Error Generation and Attribution for Multi-Agent Systems](https://mlsanity.com/paper/2509.14295/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Training Versatile Coding Agents in Synthetic Environments](https://mlsanity.com/paper/2512.12216/) (2025)
- [MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline](https://mlsanity.com/paper/2510.07307/) (2025)
- [CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges](https://mlsanity.com/paper/2603.11863/) (2026)
- [DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models](https://mlsanity.com/paper/2601.11895/) (2026)
- [BenchBench: Benchmarking Automated Benchmark Generation](https://mlsanity.com/paper/2603.20807/) (2026)
- [Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents](https://mlsanity.com/paper/2604.14004/) (2026)
- [Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests](https://mlsanity.com/paper/2606.07379/) (2026)
- [PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization](https://mlsanity.com/paper/2605.15222/) (2026)
- [Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA](https://mlsanity.com/paper/2605.29277/) (2026)
- [TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework](https://mlsanity.com/paper/2510.17163/) (2025)
- [UniCode: Augmenting Evaluation for Code Reasoning](https://mlsanity.com/paper/2510.17868/) (2025)
- [InfoSynth: Information-Guided Benchmark Synthesis for LLMs](https://mlsanity.com/paper/2601.00575/) (2026)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports -- with Preliminary Extension to Lung Cancer](https://mlsanity.com/paper/2510.23008/) (2025)
- [PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code](https://mlsanity.com/paper/2512.10713/) (2025)
- [Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?](https://mlsanity.com/paper/2601.18119/) (2026)
- [How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling](https://mlsanity.com/paper/2604.04791/) (2026)
- [Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models](https://mlsanity.com/paper/2605.18824/) (2026)
- [A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks](https://mlsanity.com/paper/2605.28556/) (2026)
- [PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization](https://mlsanity.com/paper/2607.07744/) (2026)
- [Cross-Domain Hybrid OPD for Generalizable Search Agents](https://mlsanity.com/paper/2608.02101/) (2026)
- [ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction](https://mlsanity.com/paper/2609.02067/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, with code generation emerging as a key area of focus. While numerous benchmarks have been proposed to evaluate their code generation abilities, these benchmarks face several critical limitations. First, they often rely on manual annotations, which are time-consuming and difficult to scale across different programming languages and problem complexities. Second, most existing benchmarks focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and uneven language distribution. To address these challenges, we propose AutoCodeGen, an automated method for generating high-difficulty multilingual code generation datasets without manual annotations. AutoCodeGen ensures the correctness and completeness of test cases by generating test inputs with LLMs and obtaining test outputs through a multilingual sandbox, while achieving high data quality through reverse-order problem generation and multiple filtering steps. Using this novel method, we introduce AutoCodeBench, a large-scale code generation benchmark comprising 3,920 problems evenly distributed across 20 programming languages. It is specifically designed to evaluate LLMs on challenging, diverse, and practical multilingual tasks. We evaluate over 30 leading open-source and proprietary LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The results show that even the most advanced LLMs struggle with the complexity, diversity, and multilingual nature of these tasks. Besides, we introduce AutoCodeBench-Complete, specifically designed for base models to assess their few-shot code generation capabilities. We hope the AutoCodeBench series will serve as a valuable resource and inspire the community to focus on more challenging and practical multilingual code generation scenarios.