# AixBench: A Code Generation Benchmark Dataset

arXiv: https://arxiv.org/abs/2206.13179 · HTML: https://mlsanity.com/paper/2206.13179/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2022.

## Built on by (most cited first, 22 of 22)

- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation](https://mlsanity.com/paper/2310.04951/) (2023)
- [CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation](https://mlsanity.com/paper/2311.08588/) (2023)
- [LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks](https://mlsanity.com/paper/2502.06215/) (2025)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15037/) (2024)
- [ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation](https://mlsanity.com/paper/2503.07010/) (2025)
- [AceCoder: Utilizing Existing Code to Enhance Code Generation](https://mlsanity.com/paper/2303.17780/) (2023)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)](https://mlsanity.com/paper/2409.12699/) (2024)
- [FRANC: A Lightweight Framework for High-Quality Code Generation](https://mlsanity.com/paper/2307.08220/) (2023)
- [Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark](https://mlsanity.com/paper/2412.18573/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [A Survey on Natural Language Processing for Programming](https://mlsanity.com/paper/2212.05773/) (2022)
- [Deep-Bench: Deep Learning Benchmark Dataset for Code Generation](https://mlsanity.com/paper/2502.18726/) (2025)
- [Mechanistic Understanding of Language Models in Syntactic Code Completion](https://mlsanity.com/paper/2502.18499/) (2025)
- [CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation](https://mlsanity.com/paper/2504.20673/) (2025)
- [Strategic Optimization and Challenges of Large Language Models in Object-Oriented Programming](https://mlsanity.com/paper/2408.14834/) (2024)
- [Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning](https://mlsanity.com/paper/2605.00433/) (2026)

## Abstract (arXiv)

We present a benchmark dataset for evaluating method-level code generation task. The benchmark contains a dataset of 175 samples for automated evaluation and a dataset of 161 samples for manual evaluation. We also present a new metric for automatically evaluating the correctness of the generated code, and a set of criteria to manually evaluating the overall quality of the generated code.