mlsanityData to 14 Oct 2026

Do Unlearning Methods Remove Information from Language Model Weights?

arXiv 2410.08827 ↗First posted 11 Oct 2024cs.LGcited by 73

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large Language Models' knowledge of how to perform cyber-security attacks, create bioweapons, and manipulate humans poses risks of misuse. Previous work has proposed methods to unlearn this knowledge. Historically, it has been unclear whether unlearning techniques are removing information from the model weights or just making it harder to access. To disentangle these two objectives, we propose an adversarial evaluation method to test for the removal of information from model weights: we give an attacker access to some facts that were supposed to be removed, and using those, the attacker tries to recover other facts from the same distribution that cannot be guessed from the accessible facts. We show that using fine-tuning on the accessible facts can recover 88% of the pre-unlearning accuracy when applied to current unlearning methods for information learned during pretraining, revealing the limitations of these methods in removing information from the model weights. Our results also suggest that unlearning evaluations that measure unlearning robustness on information learned during an additional fine-tuning phase may overestimate robustness compared to evaluations that attempt to unlearn information learned during pretraining.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 24 of its 32 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

24 of 32 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
  2. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Traces17 Jun
  3. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  4. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  6. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  8. 2022A Survey of Machine Unlearning6 Sep
  9. 2021Fast Yet Effective Machine Unlearning17 Nov

Other references · 15

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2020Aligning AI With Shared Human Valuescited by 723
  8. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  9. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  11. 2019On Evaluating Adversarial Robustnesscited by 447
  12. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 180
3 more
  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  2. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  3. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 37; 2026: 35.

73 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  2. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 79
  3. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  4. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 35
  5. 2025SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMscited by 22
  6. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 21
  7. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlapcited by 15
  8. 2025A General Framework to Enhance Fine-tuning-based LLM Unlearningcited by 14
  9. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  10. 2026Shaping capabilities with token-level data filteringcited by 13
  11. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarkscited by 12
  12. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 12
38 more
  1. 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularizationcited by 12
  2. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  3. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  4. 2025Distillation Robustifies Unlearningcited by 11
  5. 2025SoK: Machine Unlearning for Large Language Modelscited by 11
  6. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 11
  7. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  8. 2025Improving LLM Unlearning Robustness via Random Perturbationscited by 9
  9. 2025Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedycited by 9
  10. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  11. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 8
  12. 2025Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Modelscited by 6
  13. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 6
  14. 2025Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearningcited by 5
  15. 2025Leveraging Per-Instance Privacy for Machine Unlearningcited by 4
  16. 2025Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning Attackcited by 4
  17. 2025Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalizationcited by 4
  18. 2025Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearningcited by 4
  19. 2025A Survey on Unlearning in Large Language Modelscited by 4
  20. 2026Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignmentcited by 4
  21. 2026Gauss-Newton Unlearning for the LLM Eracited by 4
  22. 2026Is your algorithm unlearning or untraining?cited by 4
  23. 2025Access Controls Will Solve the Dual-Use Dilemmacited by 3
  24. 2025Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearningcited by 3
  25. 2026Modular Pretraining Enables Access Controlcited by 3
  26. 2025One-Point Contraction: Erasing Representational Separability toward Irreversible Deep Forgettingcited by 2
  27. 2025LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metricscited by 2
  28. 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Datacited by 2
  29. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
  30. 2025MIAT: Maneuver-Intention-Aware Transformer for Spatio-Temporal Trajectory Predictioncited by 1
  31. 2025Layered Unlearning for Adversarial Relearningcited by 1
  32. 2026Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMscited by 1
  33. 2026Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilitiescited by 1
  34. 2026Rethinking Benign Relearning: Syntax as the Hidden Driver of Unlearning Failurescited by 1
  35. 2026A Comprehensive Evaluation of LLM Unlearning Robustness under Multi-Turn Interactioncited by 1
  36. 2026Into the Gray Zone: Domain Contexts Can Blur LLM Safety Boundariescited by 1
  37. 2026Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Mattercited by 1
  38. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Auditing Language Model Unlearning via Information Decomposition
  2. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models
  3. 2025Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods
  4. 2026ROKA: Robust Knowledge Unlearning against Adversaries
  5. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning
  6. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attacks
  7. 2026Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
  8. 2025LLM Unlearning Should Be Form-Independent