mlsanityData to 7 Oct 2026

The Adverse Effects of Code Duplication in Machine Learning Models of Code

arXiv 1812.06469 ↗First posted 16 Dec 2018cs.SE cs.LGcited by 123

Not on the map. Why?

Abstract (arXiv)

The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 16; 2022: 28; 2023: 40; 2024: 23; 2025: 10; 2026: 5.

123 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  8. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  9. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  10. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  11. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  12. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
38 more
  1. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  2. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  3. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  4. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  5. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  6. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  7. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  8. 2021Self-Supervised Bug Detection and Repaircited by 59
  9. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  10. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 49
  11. 2021On the Evaluation of Neural Code Summarizationcited by 47
  12. 2023Large Language Models and Simple, Stupid Bugscited by 45
  13. 2021Multilingual training for Software Engineeringcited by 43
  14. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  15. 2024Language Models for Code Completion: A Practical Evaluationcited by 38
  16. 2021Neural Transfer Learning for Repairing Security Vulnerabilities in C Codecited by 36
  17. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  18. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  19. 2021Code Structure Guided Transformer for Source Code Summarizationcited by 29
  20. 2023CAT-LM: Training Language Models on Aligned Code And Testscited by 28
  21. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  22. 2023Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binariescited by 25
  23. 2021Project-Level Encoding for Neural Source Code Summarization of Subroutinescited by 24
  24. 2021Learning to Complete Code with Sketchescited by 23
  25. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  26. 2021Type4Py: Practical Deep Similarity Learning-Based Type Inference for Pythoncited by 19
  27. 2022Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detectioncited by 18
  28. 2024LLM4SecHW: Leveraging Domain Specific Large Language Model for Hardware Debuggingcited by 18
  29. 2021Memorization and Generalization in Neural Code Intelligence Modelscited by 17
  30. 2022Noise-Robust De-Duplication at Scalecited by 17
  31. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  32. 2023GrACE: Generation using Associated Code Editscited by 16
  33. 2023TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphscited by 16
  34. 2022On Distribution Shift in Learning-based Bug Detectorscited by 15
  35. 2023FLAME: A small language model for spreadsheet formulascited by 15
  36. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
  37. 2022Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluationcited by 14
  38. 2023Benchmarking Causal Study to Interpret Large Language Models for Source Codecited by 14

50 most cited of 123 All 123 (JSON)