# Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models

arXiv: https://arxiv.org/abs/2310.01166 · HTML: https://mlsanity.com/paper/2310.01166/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2023.

## Built on by (most cited first, 15 of 15)

- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach](https://mlsanity.com/paper/2404.14296/) (2024)
- [An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets](https://mlsanity.com/paper/2403.15230/) (2024)
- [Code Membership Inference for Detecting Unauthorized Data Use in Code Pre-trained Language Models](https://mlsanity.com/paper/2312.07200/) (2023)
- [Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code](https://mlsanity.com/paper/2402.09299/) (2024)
- [How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test](https://mlsanity.com/paper/2601.07084/) (2026)
- [Developer Perspectives on Licensing and Copyright Issues Arising from Generative AI for Software Development](https://mlsanity.com/paper/2411.10877/) (2024)
- [Effective Code Membership Inference for Code Completion Models via Adversarial Prompts](https://mlsanity.com/paper/2511.15107/) (2025)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models](https://mlsanity.com/paper/2507.17389/) (2025)
- [Exposing and Defending Membership Leakage in Vulnerability Prediction Models](https://mlsanity.com/paper/2512.08291/) (2025)
- [Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models](https://mlsanity.com/paper/2512.11482/) (2025)
- [How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?](https://mlsanity.com/paper/2512.15468/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

Given large-scale source code datasets available in open-source projects and advanced large language models, recent code models have been proposed to address a series of critical software engineering tasks, such as program repair and code completion. The training data of the code models come from various sources, not only the publicly available source code, e.g., open-source projects on GitHub but also the private data such as the confidential source code from companies, which may contain sensitive information (for example, SSH keys and personal information). As a result, the use of these code models may raise new privacy concerns. In this paper, we focus on a critical yet not well-explored question on using code models: what is the risk of membership information leakage in code models? Membership information leakage refers to the risk that an attacker can infer whether a given data point is included in (i.e., a member of) the training data. To answer this question, we propose Gotcha, a novel membership inference attack method specifically for code models. We investigate the membership leakage risk of code models. Our results reveal a worrying fact that the risk of membership leakage is high: although the previous attack methods are close to random guessing, Gotcha can predict the data membership with a high true positive rate of 0.95 and a low false positive rate of 0.10. We also show that the attacker's knowledge of the victim model (e.g., the model architecture and the pre-training data) impacts the success rate of attacks. Further analysis demonstrates that changing the decoding strategy can mitigate the risk of membership leakage. This study calls for more attention to understanding the privacy of code models and developing more effective countermeasures against such attacks.