mlsanityData to 7 Oct 2026

Rethinking LLM Memorization through the Lens of Adversarial Compression

arXiv 2404.15146 ↗First posted 23 Apr 2024cs.LG cs.CLcited by 91

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all their training data or they integrate many data sources in some way more akin to how a human would learn and synthesize information. The answer hinges, to a large degree, on how we define memorization. In this work, we propose the Adversarial Compression Ratio (ACR) as a metric for assessing memorization in LLMs. A given string from the training data is considered memorized if it can be elicited by a prompt (much) shorter than the string itself -- in other words, if these strings can be "compressed" with the model by computing adversarial prompts of fewer tokens. The ACR overcomes the limitations of existing notions of memorization by (i) offering an adversarial view of measuring memorization, especially for monitoring unlearning and compliance; and (ii) allowing for the flexibility to measure memorization for arbitrary strings at a reasonably low compute. Our definition serves as a practical tool for determining when model owners may be violating terms around data usage, providing a potential legal tool and a critical lens through which to address such scenarios.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 40 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

21 of 40 references.

References · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  6. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  7. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  8. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  9. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  10. 2023Language Modeling Is Compressioncited by 225
  11. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  12. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
9 more
  1. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  2. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  3. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  4. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  5. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  6. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  7. 2024Robust Concept Erasure Using Task Vectorscited by 27
  8. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  9. 2023Report of the 1st Workshop on Generative AI and Lawcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 48; 2026: 15.

91 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  4. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  5. 2025How much do language models memorize?cited by 53
  6. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  7. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  8. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  9. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  10. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  11. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  12. 2025A Survey on Data Contamination for Large Language Modelscited by 33
38 more
  1. 2024Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenoncited by 30
  2. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  3. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  5. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  6. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 16
  7. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 16
  8. 2026Extracting books from production language modelscited by 16
  9. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  10. 2024Mitigating Memorization In Language Modelscited by 11
  11. 2024The Mosaic Memory of Large Language Modelscited by 9
  12. 2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
  13. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 9
  14. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  15. 2025UCD: Unlearning in LLMs via Contrastive Decodingcited by 9
  16. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  17. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  18. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  19. 2024Attributing Culture-Conditioned Generations to Pretraining Corporacited by 8
  20. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 7
  21. 2025SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domainscited by 7
  22. 2025Extracting alignment data in open modelscited by 7
  23. 2024Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMscited by 6
  24. 2025Mitigating Unintended Memorization with LoRA in Federated Learning for LLMscited by 6
  25. 2025Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models via Automated Adversarial Promptingcited by 6
  26. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  27. 2025On the Impossibility of Retrain Equivalence in Machine Unlearningcited by 6
  28. 2024Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizerscited by 5
  29. 2024Order of Magnitude Speedups for LLM Membership Inferencecited by 5
  30. 2024Evil twins are not that evil: Qualitative insights into machine-generated promptscited by 5
  31. 2025Style over Substance: Distilled Language Models Reason Via Stylistic Replicationcited by 5
  32. 2025Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysiscited by 5
  33. 2025QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluationcited by 5
  34. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  35. 2024Training Foundation Models as Data Compression: On Information, Model Weights and Copyright Lawcited by 4
  36. 2025Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Modelscited by 4
  37. 2025PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Modelscited by 3
  38. 2025Empirical Privacy Variancecited by 3

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attribution
  2. 2023SoK: Memorization in General-Purpose Large Language Models
  3. 2024Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
  4. 2026Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
  5. 2025Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers
  6. 2025Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities
  7. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMs
  8. 2026Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors