mlsanityData to 14 Oct 2026

CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit

arXiv 2411.10842 ↗First posted 16 Nov 2024cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Data contamination presents a critical barrier preventing widespread industrial adoption of advanced software engineering techniques that leverage code language models (CLMs). This phenomenon occurs when evaluation data inadvertently overlaps with the public code repositories used to train CLMs, severely undermining the credibility of performance evaluations. For software companies considering the integration of CLM-based techniques into their development pipeline, this uncertainty about true performance metrics poses an unacceptable business risk. Code refactoring, which comprises code restructuring and variable renaming, has emerged as a promising measure to mitigate data contamination. It provides a practical alternative to the resource-intensive process of building contamination-free evaluation datasets, which would require companies to collect, clean, and label code created after the CLMs' training cutoff dates. However, the lack of automated code refactoring tools and scientifically validated refactoring techniques has hampered widespread industrial implementation. To bridge the gap, this paper presents the first systematic study to examine the efficacy of code refactoring operators at multiple scales (method-level, class-level, and cross-class level) and in different programming languages. In particular, we develop an open-sourced toolkit, CODECLEANER, which includes 11 operators for Python, with nine method-level, one class-level, and one cross-class-level operator. A drop of 65% overlap ratio is found when applying all operators in CODECLEANER, demonstrating their effectiveness in addressing data contamination. Additionally, we migrate four operators to Java, showing their generalizability to another language. We make CODECLEANER online available to facilitate further studies on mitigating CLM data contamination.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 49 of its 70 references that are in the map. First posted on arXiv 16 Nov 2024.

Builds on

49 of 70 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
37 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  2. 2023StarCoder: may the source be with you!cited by 799
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  7. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  9. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  10. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  11. 2022Data Contamination: From Memorization to Exploitationcited by 236
  12. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  13. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  14. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  15. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  16. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  17. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  18. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  19. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  20. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  21. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
  22. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  23. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  24. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  25. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 71
  26. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 52
  27. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  28. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  29. 2023Data Portraits: Recording Foundation Model Training Datacited by 30
  30. 2019Structure-Invariant Testing for Machine Translationcited by 28
  31. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  32. 2024Exploring Multi-Lingual Bias of Large Code Models in Code Generationcited by 17
  33. 2024Concerned with Data Contamination? Assessing Countermeasures in Code Language Modelcited by 14
  34. 2024MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testingcited by 6
  35. 2020SemMT: A Semantic-based Testing Approach for Machine Translation Systemscited by 5
  36. 2023Word Closure-Based Metamorphic Testing for Machine Translationcited by 4
  37. 2023Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippetscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  2. 2025Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generationcited by 3