mlsanityData to 7 Oct 2026

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

arXiv 2403.04811 ↗First posted 6 Mar 2024cs.SE cs.CL cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

While large language models have achieved remarkable performance on various code generation benchmarks, there have been growing concerns regarding potential contamination of these benchmarks as they may be leaked into pretraining and finetuning data. While recent work has investigated contamination in natural language generation and understanding tasks, there has been less extensive research into how data contamination impacts the evaluation of code generation, which is critical for understanding the robustness and reliability of LLMs in programming contexts. In this work, we perform a comprehensive study of data contamination of popular code generation benchmarks, and precisely quantify their overlap with pretraining corpus through both surface-level and semantic-level matching. In our experiments, we show that there are substantial overlap between popular code generation benchmarks and open training corpus, and models perform significantly better on the subset of the benchmarks where similar solutions are seen during training. We also conduct extensive analysis on the factors that affects model memorization and generalization, such as model size, problem difficulty, and question length. We release all resulting files from our matching pipeline for future research.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

23 of 41 references.

References · 23

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
11 more
  1. 2022Data Contamination: From Memorization to Exploitationcited by 234
  2. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  3. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  4. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  5. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  6. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  7. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  8. 2020Understanding Unintended Memorization in Federated Learningcited by 36
  9. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  10. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  11. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 15; 2026: 29.

62 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  3. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  4. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  5. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  6. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 38
  7. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  8. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  9. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  10. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  11. 2024VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbationcited by 29
  12. 2024On Leakage of Code Generation Evaluation Datasetscited by 29
38 more
  1. 2024Data Contamination Can Cross Language Barrierscited by 26
  2. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  3. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  4. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 19
  5. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  6. 2024SEC-QA: A Systematic Evaluation Corpus for Financial QAcited by 17
  7. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  8. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  9. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  10. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 9
  11. 2026Soft Contamination Means Benchmarks Test Shallow Generalizationcited by 8
  12. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Codecited by 7
  13. 2025Memorize or Generalize? Evaluating LLM Code Generation with Code Rewritingcited by 6
  14. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  15. 2026Coding Agents Don't Know When to Actcited by 4
  16. 2024Questionable practices in machine learningcited by 3
  17. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  18. 2025Unbiased Evaluation of Large Language Models from a Causal Perspectivecited by 3
  19. 2025Trust Me, I Know This Function: Hijacking LLM Static Analysis using Biascited by 3
  20. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 3
  21. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Timecited by 3
  22. 2025NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Taskscited by 2
  23. 2026Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysiscited by 2
  24. 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational studycited by 2
  25. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  26. 2025UniCode: Augmenting Evaluation for Code Reasoningcited by 1
  27. 2026Quantifying the Effect of Test Set Contamination on Generative Evaluationscited by 1
  28. 2026Pull Requests as a Training Signal for Repo-Level Code Editingcited by 1
  29. 2026SecCodeBench-V2 Technical Reportcited by 1
  30. 2026TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMscited by 1
  31. 2026Vero: Can AI Agents Build Formally Verified Software Repositories?cited by 1
  32. 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 1
  33. 2025Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligencecited by 0
  34. 2026LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDBcited by 0
  35. 2026Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesiscited by 0
  36. 2026ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coveragecited by 0
  37. 2026Self-Refining Topology Optimization via an LLM-Based Multi-Agent Frameworkcited by 0
  38. 2026SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agentscited by 0

50 most cited of 62 All 62 (JSON)