# Rethinking LLM Memorization through the Lens of Adversarial Compression

arXiv: https://arxiv.org/abs/2404.15146 · HTML: https://mlsanity.com/paper/2404.15146/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 21 of 21)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2310.15140/) (2023)
- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Robust Concept Erasure Using Task Vectors](https://mlsanity.com/paper/2404.03631/) (2024)
- [Grokking as Compression: A Nonlinear Complexity Perspective](https://mlsanity.com/paper/2310.05918/) (2023)
- [Report of the 1st Workshop on Generative AI and Law](https://mlsanity.com/paper/2311.06477/) (2023)

## Built on by (most cited first, 30 of 94)

- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs](https://mlsanity.com/paper/2407.15549/) (2024)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [How much do language models memorize?](https://mlsanity.com/paper/2505.24832/) (2025)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs](https://mlsanity.com/paper/2406.10209/) (2024)
- [Safety Pretraining: Toward the Next Generation of Safe AI](https://mlsanity.com/paper/2504.16980/) (2025)
- [Measuring memorization in language models via probabilistic extraction](https://mlsanity.com/paper/2410.19482/) (2024)
- [CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation](https://mlsanity.com/paper/2407.07087/) (2024)
- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation](https://mlsanity.com/paper/2406.12975/) (2024)
- [Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon](https://mlsanity.com/paper/2406.17746/) (2024)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions](https://mlsanity.com/paper/2410.18966/) (2024)
- [A Probabilistic Perspective on Unlearning and Alignment for Large Language Models](https://mlsanity.com/paper/2410.03523/) (2024)
- [The Pitfalls of Memorization: When Memorization Hurts Generalization](https://mlsanity.com/paper/2412.07684/) (2024)
- [Quantifying Generalization Complexity for Large Language Models](https://mlsanity.com/paper/2410.01769/) (2024)
- [Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models](https://mlsanity.com/paper/2405.05990/) (2024)
- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [Mitigating Memorization In Language Models](https://mlsanity.com/paper/2410.02159/) (2024)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [SoK: Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2506.09227/) (2025)
- [The Mosaic Memory of Large Language Models](https://mlsanity.com/paper/2405.15523/) (2024)
- [Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training](https://mlsanity.com/paper/2502.15680/) (2025)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Model](https://mlsanity.com/paper/2502.11169/) (2025)
- [UCD: Unlearning in LLMs via Contrastive Decoding](https://mlsanity.com/paper/2506.12097/) (2025)
- [Towards More Realistic Extraction Attacks: An Adversarial Perspective](https://mlsanity.com/paper/2407.02596/) (2024)
- [Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2408.04643/) (2024)

## Abstract (arXiv)

Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on how we define memorization. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs. A given string from the training data is considered memorized if it can be elicited by a prompt (much) shorter than the string itself -- in other words, if these strings can be "compressed" with the model by computing adversarial prompts of fewer tokens. The ACR overcomes the limitations of existing notions of memorization by (i) offering an adversarial view of measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute. Our definition serves as a practical tool for determining when model owners may be violating terms around data usage, providing a potential legal tool and a critical lens through which to address such scenarios.