# Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

arXiv: https://arxiv.org/abs/2501.10711 · HTML: https://mlsanity.com/paper/2501.10711/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2025.

## Built on by (most cited first, 10 of 10)

- [RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback](https://mlsanity.com/paper/2510.06186/) (2025)
- [RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation](https://mlsanity.com/paper/2503.16922/) (2025)
- [Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation](https://mlsanity.com/paper/2506.03535/) (2025)
- [Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space](https://mlsanity.com/paper/2601.06220/) (2026)
- [Re-Evaluating Code LLM Benchmarks Under Semantic Mutation](https://mlsanity.com/paper/2506.17369/) (2025)
- [Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs](https://mlsanity.com/paper/2511.02197/) (2025)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation](https://mlsanity.com/paper/2607.00062/) (2026)
- [TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution](https://mlsanity.com/paper/2610.10619/) (2026)

## Abstract (arXiv)

Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.