mlsanityData to 7 Oct 2026

TOFU: A Task of Fictitious Unlearning for LLMs

arXiv 2401.06121 ↗First posted 11 Jan 2024cs.LG cs.CLcited by 507

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or tuning models to forget information present in their training data, provides us with a way to protect private data after training. Although several methods exist for such unlearning, it is unclear to what extent they result in models equivalent to those where the data to be forgotten was never learned in the first place. To address this challenge, we present TOFU, a Task of Fictitious Unlearning, as a benchmark aimed at helping deepen our understanding of unlearning. We offer a dataset of 200 diverse synthetic author profiles, each consisting of 20 question-answer pairs, and a subset of these profiles called the forget set that serves as the target for unlearning. We compile a suite of metrics that work together to provide a holistic picture of unlearning efficacy. Finally, we provide a set of baseline results from existing unlearning algorithms. Importantly, none of the baselines we consider show effective unlearning motivating continued efforts to develop approaches for unlearning that effectively tune models so that they truly behave as if they were never trained on the forget data at all.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 38 of its 49 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

38 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  2. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  3. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation16 Aug
  4. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions8 Jul
  5. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment11 May
  6. 2023Towards Unbounded Machine Unlearning20 Feb
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
  9. 2022Continual Learning and Private Unlearning24 Mar
  10. 2022Towards Adversarial Evaluations for Inexact Machine Unlearning17 Jan
  11. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct
  12. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  10. 2019Machine Unlearningcited by 1,020
  11. 2021Membership Inference Attacks From First Principlescited by 839
  12. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
14 more
  1. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  2. 2021Editing Factual Knowledge in Language Modelscited by 548
  3. 2019Certified Data Removal from Machine Learning Modelscited by 512
  4. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  5. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  6. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  7. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  8. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  9. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  10. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  11. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  12. 2023Privacy Auditing with One (1) Training Runcited by 119
  13. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  14. 2023On Provable Copyright Protection for Generative Modelscited by 97

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 97; 2025: 184; 2026: 226.

507 papers.

Sort:

By area:

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  2. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  7. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  8. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  9. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  10. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  11. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  12. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
38 more
  1. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  2. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  3. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  4. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  5. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  6. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  7. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  8. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  9. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  10. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  11. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  12. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 59
  13. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  14. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  15. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  16. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 54
  17. 2024On Large Language Model Continual Unlearningcited by 51
  18. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  19. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  20. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  21. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  22. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  23. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  24. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  25. 2024Soft Prompting for Unlearning in Large Language Modelscited by 42
  26. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  27. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  28. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  29. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  30. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  31. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  32. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  33. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 36
  34. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  35. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  36. 2024Offset Unlearning for Large Language Modelscited by 33
  37. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
  38. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Modelscited by 32

50 most cited of 507 All 507 (JSON)