# On Leakage of Code Generation Evaluation Datasets

arXiv: https://arxiv.org/abs/2407.07565 · HTML: https://mlsanity.com/paper/2407.07565/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models](https://mlsanity.com/paper/2403.04811/) (2024)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks](https://mlsanity.com/paper/2406.15325/) (2024)

## Built on by (most cited first, 30 of 32)

- [Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation](https://mlsanity.com/paper/2412.15118/) (2024)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [Learning Code Preference via Synthetic Evolution](https://mlsanity.com/paper/2410.03837/) (2024)
- [CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs](https://mlsanity.com/paper/2410.01999/) (2024)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code](https://mlsanity.com/paper/2503.00691/) (2025)
- [RepairBench: Leaderboard of Frontier Models for Program Repair](https://mlsanity.com/paper/2409.18952/) (2024)
- [Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts](https://mlsanity.com/paper/2408.15901/) (2024)
- [Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems](https://mlsanity.com/paper/2410.00151/) (2024)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Disproving Program Equivalence with LLMs](https://mlsanity.com/paper/2502.18473/) (2025)
- [Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis](https://mlsanity.com/paper/2505.03019/) (2025)
- [Statistical Methods in Generative AI](https://mlsanity.com/paper/2509.07054/) (2025)
- [BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks](https://mlsanity.com/paper/2608.30724/) (2026)
- [Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis](https://mlsanity.com/paper/2506.04142/) (2025)
- [LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering](https://mlsanity.com/paper/2608.28281/) (2026)
- [CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit](https://mlsanity.com/paper/2411.10842/) (2024)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis](https://mlsanity.com/paper/2604.24703/) (2026)
- [Efficient Benchmarking Is Just Feature Selection and Multiple Regression](https://mlsanity.com/paper/2605.25773/) (2026)
- [Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models](https://mlsanity.com/paper/2606.16999/) (2026)
- [Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study](https://mlsanity.com/paper/2607.02436/) (2026)
- [Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task Success at Scale](https://mlsanity.com/paper/2505.13511/) (2025)
- [Verification Limits Code LLM Training](https://mlsanity.com/paper/2509.20837/) (2025)
- [Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners](https://mlsanity.com/paper/2511.05501/) (2025)
- [SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents](https://mlsanity.com/paper/2606.18733/) (2026)
- [SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models](https://mlsanity.com/paper/2606.29815/) (2026)
- [Memorization Diagnostics for Code LLMs Should be Scale-Aware](https://mlsanity.com/paper/2608.12771/) (2026)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp .