mlsanityData to 14 Oct 2026

Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

arXiv 2507.17389 ↗First posted 23 Jul 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

31 of 50 references.

References · 31

  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023StarCoder: may the source be with you!cited by 799
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  11. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  12. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
19 more
  1. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  2. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  3. 2023SantaCoder: don't reach for the stars!cited by 170
  4. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  5. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  6. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  7. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64
  8. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 55
  9. 2023Traces of Memorisation in Large Language Models for Codecited by 29
  10. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  11. 2024Probing Language Models for Pre-training Data Detectioncited by 25
  12. 2023Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Modelscited by 15
  13. 2022To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?cited by 11
  14. 2024Does Your Neural Code Completion Model Use My Code? A Membership Inference Approachcited by 11
  15. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  16. 2024On Inter-dataset Code Duplication and Data Leakage in Large Language Modelscited by 10
  17. 2024ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Modelscited by 10
  18. 2023Code Membership Inference for Detecting Unauthorized Data Use in Code Pre-trained Language Modelscited by 8
  19. 2024Towards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Frameworkcited by 4

Built on by

Not cited yet.