# MoPe: Model Perturbation-based Privacy Attacks on Language Models

arXiv: https://arxiv.org/abs/2310.14369 · HTML: https://mlsanity.com/paper/2310.14369/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Enhanced Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/2111.09679/) (2021)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Large-Scale Differentially Private BERT](https://mlsanity.com/paper/2108.01624/) (2021)
- [Measuring Forgetting of Memorized Training Examples](https://mlsanity.com/paper/2207.00099/) (2022)
- [On Provable Copyright Protection for Generative Models](https://mlsanity.com/paper/2302.10870/) (2023)
- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)
- [Scalable Membership Inference Attacks via Quantile Regression](https://mlsanity.com/paper/2307.03694/) (2023)
- [Membership Inference Attack Susceptibility of Clinical Language Models](https://mlsanity.com/paper/2104.08305/) (2021)
- [Understanding Unintended Memorization in Federated Learning](https://mlsanity.com/paper/2006.07490/) (2020)
- [On the Privacy Risks of Algorithmic Recourse](https://mlsanity.com/paper/2211.05427/) (2022)
- [Copyright in Generative Deep Learning](https://mlsanity.com/paper/2105.09266/) (2021)
- [How BPE Affects Memorization in Transformers](https://mlsanity.com/paper/2110.02782/) (2021)
- [Membership Inference Attack Using Self Influence Functions](https://mlsanity.com/paper/2205.13680/) (2022)

## Built on by (most cited first, 25 of 25)

- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Blind Baselines Beat Membership Inference Attacks for Foundation Models](https://mlsanity.com/paper/2406.16201/) (2024)
- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [Critical windows: non-asymptotic theory for feature emergence in diffusion models](https://mlsanity.com/paper/2403.01633/) (2024)
- [Copyright Traps for Large Language Models](https://mlsanity.com/paper/2402.09363/) (2024)
- [Membership Inference Attacks Against Vision-Language Models](https://mlsanity.com/paper/2501.18624/) (2025)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Language Models May Verbatim Complete Text They Were Not Explicitly Trained On](https://mlsanity.com/paper/2503.17514/) (2025)
- [The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text](https://mlsanity.com/paper/2502.14921/) (2025)
- [Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2405.08597/) (2024)
- [A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models](https://mlsanity.com/paper/2311.11415/) (2023)
- [Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models](https://mlsanity.com/paper/2402.17012/) (2024)
- [Do Parameters Reveal More than Loss for Membership Inference?](https://mlsanity.com/paper/2406.11544/) (2024)
- [LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models](https://mlsanity.com/paper/2507.18302/) (2025)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning](https://mlsanity.com/paper/2506.14913/) (2025)
- [Auditing Training Data in Domain-adapted LLMs: LoRA-MINT](https://mlsanity.com/paper/2606.06946/) (2026)
- [Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Models](https://mlsanity.com/paper/2503.01208/) (2025)
- [Membership and Memorization in LLM Knowledge Distillation](https://mlsanity.com/paper/2508.07054/) (2025)
- [CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models](https://mlsanity.com/paper/2606.17464/) (2026)
- [Initial Exploration of Zero-Shot Privacy Utility Tradeoffs in Tabular Data Using GPT-4](https://mlsanity.com/paper/2404.05047/) (2024)
- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [Counterfactual Influence as a Distributional Quantity](https://mlsanity.com/paper/2506.20481/) (2025)
- [Leak It: Per-Document Extraction Beyond Aggregate Membership Inference](https://mlsanity.com/paper/2608.00144/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)

## Abstract (arXiv)

Recent work has shown that Large Language Models (LLMs) can unintentionally leak sensitive information present in their training data. In this paper, we present Model Perturbations (MoPe), a new method to identify with high confidence if a given text is in the training data of a pre-trained language model, given white-box access to the models parameters. MoPe adds noise to the model in parameter space and measures the drop in log-likelihood at a given point x, a statistic we show approximates the trace of the Hessian matrix with respect to model parameters. Across language models ranging from 70M to 12B parameters, we show that MoPe is more effective than existing loss-based attacks and recently proposed perturbation-based methods. We also examine the role of training point order and model size in attack success, and empirically demonstrate that MoPe accurately approximate the trace of the Hessian in practice. Our results show that the loss of a point alone is insufficient to determine extractability -- there are training points we can recover using our method that have average loss. This casts some doubt on prior works that use the loss of a point as evidence of memorization or unlearning.