mlsanityData to 7 Oct 2026

On Leakage of Code Generation Evaluation Datasets

arXiv 2407.07565 ↗First posted 10 Jul 2024cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models. We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp .

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

17 of 24 references.

References · 17

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2023StarCoder: may the source be with you!cited by 790
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 354
5 more
  1. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  2. 2023What's In My Big Data?cited by 129
  3. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  4. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  5. 2024Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stackscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 12.

29 papers.

Sort:

By area:

  1. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  2. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  3. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  4. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  5. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  6. 2024RepairBench: Leaderboard of Frontier Models for Program Repaircited by 9
  7. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  8. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
  9. 2024Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problemscited by 6
  10. 2025Disproving Program Equivalence with LLMscited by 5
  11. 2025Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysiscited by 5
  12. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
17 more
  1. 2025Statistical Methods in Generative AIcited by 4
  2. 2025Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysiscited by 3
  3. 2026BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Taskscited by 3
  4. 2024CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkitcited by 2
  5. 2026Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysiscited by 2
  6. 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational studycited by 2
  7. 2025Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task Success at Scalecited by 1
  8. 2025Verification Limits Code LLM Trainingcited by 1
  9. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 1
  10. 2026Efficient Benchmarking Is Just Feature Selection and Multiple Regressioncited by 1
  11. 2026Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Modelscited by 1
  12. 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 1
  13. 2025Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitionerscited by 0
  14. 2026SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agentscited by 0
  15. 2026SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Modelscited by 0
  16. 2026Memorization Diagnostics for Code LLMs Should be Scale-Awarecited by 0
  17. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0