# Traces of Memorisation in Large Language Models for Code

arXiv: https://arxiv.org/abs/2312.11658 · HTML: https://mlsanity.com/paper/2312.11658/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [Reconstruction and Membership Inference Attacks against Generative Models](https://mlsanity.com/paper/1906.03006/) (2019)
- [CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation](https://mlsanity.com/paper/2206.06888/) (2022)
- [Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?](https://mlsanity.com/paper/1904.05506/) (2019)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Training Data Extraction From Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2305.16157/) (2023)

## Built on by (most cited first, 29 of 29)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [LiCoEval: Evaluating LLMs on License Compliance in Code Generation](https://mlsanity.com/paper/2408.02487/) (2024)
- [API-guided Dataset Synthesis to Finetune Large Code Models](https://mlsanity.com/paper/2408.08343/) (2024)
- [An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets](https://mlsanity.com/paper/2403.15230/) (2024)
- [Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting](https://mlsanity.com/paper/2503.02296/) (2025)
- [Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning](https://mlsanity.com/paper/2509.13755/) (2025)
- [Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs](https://mlsanity.com/paper/2506.17353/) (2025)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [CodeCipher: Learning to Obfuscate Source Code Against LLMs](https://mlsanity.com/paper/2410.05797/) (2024)
- [How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning](https://mlsanity.com/paper/2501.17501/) (2025)
- [Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships](https://mlsanity.com/paper/2404.19432/) (2024)
- [Low-Perplexity LLM-Generated Sequences and Where To Find Them](https://mlsanity.com/paper/2507.01844/) (2025)
- [Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization](https://mlsanity.com/paper/2510.16022/) (2025)
- [Effective Code Membership Inference for Code Completion Models via Adversarial Prompts](https://mlsanity.com/paper/2511.15107/) (2025)
- [Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs](https://mlsanity.com/paper/2601.10496/) (2026)
- [The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark](https://mlsanity.com/paper/2608.11469/) (2026)
- [Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation](https://mlsanity.com/paper/2503.22760/) (2025)
- [Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligence](https://mlsanity.com/paper/2506.02791/) (2025)
- [Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models](https://mlsanity.com/paper/2507.17389/) (2025)
- [DevLicOps: A Framework for Mitigating Licensing Risks in AI-Generated Code](https://mlsanity.com/paper/2508.16853/) (2025)
- [Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models](https://mlsanity.com/paper/2512.11482/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models](https://mlsanity.com/paper/2601.02002/) (2026)
- [SERSEM: Selective Entropy-Weighted Scoring for Membership Inference in Code Language Models](https://mlsanity.com/paper/2604.01147/) (2026)
- [A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair](https://mlsanity.com/paper/2604.21579/) (2026)
- [Feedback-Normalized Developer Memory for Reinforcement-Learning Coding Agents: A Safety-Gated MCP Architecture](https://mlsanity.com/paper/2605.01567/) (2026)
- [Characterizing the Quality Profile of AI-Generated C++ in Production](https://mlsanity.com/paper/2608.06640/) (2026)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

Large language models have gained significant popularity because of their ability to generate human-like text and potential applications in various fields, such as Software Engineering. Large language models for code are commonly trained on large unsanitised corpora of source code scraped from the internet. The content of these datasets is memorised and can be extracted by attackers with data extraction attacks. In this work, we explore memorisation in large language models for code and compare the rate of memorisation with large language models trained on natural language. We adopt an existing benchmark for natural language and construct a benchmark for code by identifying samples that are vulnerable to attack. We run both benchmarks against a variety of models, and perform a data extraction attack. We find that large language models for code are vulnerable to data extraction attacks, like their natural language counterparts. From the training data that was identified to be potentially extractable we were able to extract 47% from a CodeGen-Mono-16B code completion model. We also observe that models memorise more, as their parameter count grows, and that their pre-training data are also vulnerable to attack. We also find that data carriers are memorised at a higher rate than regular code or documentation and that different model architectures memorise different samples. Data leakage has severe outcomes, so we urge the research community to further investigate the extent of this phenomenon using a wider range of models and extraction techniques in order to build safeguards to mitigate this issue.