Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
Not on the map. Why?
Abstract (arXiv)
The evaluation of code-generating Large Language Models (LLMs) is fundamentally constrained by two intertwined challenges: a reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing. This paper introduces a new benchmark construction philosophy, Dual Scaling, designed to systematically address both limitations. Our approach involves continuously scaling the source of problems from dynamic, real-world code repositories and systematically scaling the rigor of tests via automated, high-coverage Property-Based Testing (PBT). We instantiate this philosophy in CODE2BENCH, an end-to-end framework that leverages Scope Graph analysis for principled dependency classification and a 100% branch coverage quality gate to ensure test suite integrity. Using this framework, we construct CODE2BENCH-2509, a new benchmark suite with native instances in both Python and Java. Our extensive evaluation of 10 state-of-the-art LLMs on CODE2BENCH-2509, powered by a novel "diagnostic fingerprint" visualization, yields three key insights: (1) models exhibit a fundamental performance gap, excelling at API application (Weakly Self-Contained tasks) but struggling with algorithmic synthesis (Self-Contained tasks); (2) a model's performance is profoundly shaped by the target language's ecosystem, a nuance we are the first to systematically quantify; and (3) our rigorous, scaled testing is critical in uncovering an "illusion of correctness" prevalent in simpler benchmarks. Our work presents a robust, scalable, and diagnostic paradigm for the next generation of LLM evaluation in software engineering. The code, data, and results are available at https://code2bench.github.io/.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 10 Aug 2025.
Builds on
35 of 47 references.
References · 35
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
23 more
- 2021Measuring Coding Challenge Competence With APPScited by 902
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
- 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
- 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
- 2019SPoC: Search-based Pseudocode to Codecited by 176
- 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
- 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 73
- 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
- 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 65
- 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
- 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 40
- 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
- 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 34
- 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
- 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
- 2024OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Modelscited by 8
Built on by
Citing papers per year: 2026: 3.
3 papers.