mlsanityData to 14 Oct 2026

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

arXiv 2404.05868 ↗First posted 8 Apr 2024cs.LG cs.AI cs.CLcited by 504

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse -- a drastic degradation of the model's utilities. In this paper, we propose Negative Preference Optimization (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data.

Key facts. Cited by 504 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 44 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

36 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2019Machine Unlearningcited by 1,061
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  11. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  12. 2019Certified Data Removal from Machine Learning Modelscited by 535
23 more
  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  3. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  4. 2022Memory-Based Model Editing at Scalecited by 447
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  7. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  8. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 371
  9. 2022A Survey of Machine Unlearningcited by 312
  10. 2023Large Language Model Unlearningcited by 302
  11. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 279
  12. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  13. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  14. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  15. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  16. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  17. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  18. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  19. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  20. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 119
  21. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 99
  22. 2023Artificial intelligence and biological misuse: Differentiating risks of language models and biological design toolscited by 45
  23. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 81; 2025: 192; 2026: 231.

504 papers.

Sort:

By area:

  1. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 261
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  5. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  7. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  8. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  10. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  11. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  12. 2024Learning Dynamics of LLM Finetuningcited by 95
38 more
  1. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 88
  2. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 84
  3. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  4. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 81
  5. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 79
  6. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
  7. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 75
  8. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  9. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 64
  10. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  11. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 61
  12. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  13. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 56
  14. 2024On Large Language Model Continual Unlearningcited by 52
  15. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  16. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  17. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  18. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  19. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  20. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
  21. 2024Soft Prompting for Unlearning in Large Language Modelscited by 46
  22. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  23. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 45
  24. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  25. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 43
  26. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 42
  27. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  28. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 40
  29. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 39
  30. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  31. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 36
  32. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  33. 2025A Survey of Direct Preference Optimizationcited by 36
  34. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 34
  35. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Modelscited by 33
  36. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  37. 2024On Effects of Steering Latent Representation for Large Language Model Unlearningcited by 32
  38. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 31

50 most cited of 504 All 504 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Objective Large Language Model Unlearning
  2. 2024Machine Unlearning in Large Language Models
  3. 2026Explainable LLM Unlearning Through Reasoning
  4. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
  5. 2025Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
  6. 2024Offset Unlearning for Large Language Models
  7. 2024Mitigating Social Biases in Language Models through Unlearning
  8. 2026Unbiased Alignment for Large Language Models with Noisy Preferences