# Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach

arXiv: https://arxiv.org/abs/2404.14296 · HTML: https://mlsanity.com/paper/2404.14296/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Systematic Evaluation of Privacy Risks of Machine Learning Models](https://mlsanity.com/paper/2003.10595/) (2020)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [A survey of deep neural network watermarking techniques](https://mlsanity.com/paper/2103.09274/) (2021)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Code Completion with Neural Attention and Pointer Networks](https://mlsanity.com/paper/1711.09573/) (2017)
- [CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning](https://mlsanity.com/paper/2110.12925/) (2021)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)
- [FakeTagger: Robust Safeguards against DeepFake Dissemination via Provenance Tracking](https://mlsanity.com/paper/2009.09869/) (2020)
- [Digital Watermarking for Deep Neural Networks](https://mlsanity.com/paper/1802.02601/) (2018)
- [CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models](https://mlsanity.com/paper/2308.14401/) (2023)
- [Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs](https://mlsanity.com/paper/2103.09499/) (2021)

## Built on by (most cited first, 11 of 11)

- [NES: An Instruction-Free, Low-Latency Next Edit Suggestion Framework Powered by Learned Historical Editing Trajectories](https://mlsanity.com/paper/2508.02473/) (2025)
- [Effective Code Membership Inference for Code Completion Models via Adversarial Prompts](https://mlsanity.com/paper/2511.15107/) (2025)
- [Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs](https://mlsanity.com/paper/2601.10496/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications](https://mlsanity.com/paper/2605.26133/) (2026)
- [code_transformed: The Influence of Large Language Models on Code](https://mlsanity.com/paper/2506.12014/) (2025)
- [Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models](https://mlsanity.com/paper/2507.17389/) (2025)
- [Exposing and Defending Membership Leakage in Vulnerability Prediction Models](https://mlsanity.com/paper/2512.08291/) (2025)
- [Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models](https://mlsanity.com/paper/2512.11482/) (2025)
- [AST-PAC: AST-guided Membership Inference for Code](https://mlsanity.com/paper/2602.13240/) (2026)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

Recent years have witnessed significant progress in developing deep learning-based models for automated code completion. Although using source code in GitHub has been a common practice for training deep-learning-based models for code completion, it may induce some legal and ethical issues such as copyright infringement. In this paper, we investigate the legal and ethical issues of current neural code completion models by answering the following question: Is my code used to train your neural code completion model? To this end, we tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks to a more challenging task of code completion. In particular, since the target code completion models perform as opaque black boxes, preventing access to their training data and parameters, we opt to train multiple shadow models to mimic their behavior. The acquired posteriors from these shadow models are subsequently employed to train a membership classifier. Subsequently, the membership classifier can be effectively employed to deduce the membership status of a given code sample based on the output of a target code completion model. We comprehensively evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models, (i.e., LSTM-based, CodeGPT, CodeGen, and StarCoder). Experimental results reveal that the LSTM-based and CodeGPT models suffer the membership leakage issue, which can be easily detected by our proposed membership inference approach with an accuracy of 0.842, and 0.730, respectively. Interestingly, our experiments also show that the data membership of current large language models of code, e.g., CodeGen and StarCoder, is difficult to detect, leaving ampler space for further improvement. Finally, we also try to explain the findings from the perspective of model memorization.