mlsanityData to 7 Oct 2026

Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning

arXiv 2509.22263 ↗First posted 26 Sep 2025cs.LGcited by 5

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks. Although some unlearning methods mitigate these risks, they remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface. In this paper, we show that widely used unlearning methods cause shallow alignment: instead of faithfully erasing target knowledge, they generate spurious unlearning neurons that amplify negative influence to hide it. To overcome this limitation, we introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge. Experimental results confirm that our method reliably erases target knowledge and outperforms strong baselines across two practical retraining scenarios: (1) adversarial injection of private data, and (2) benign attack using an instruction-following benchmark. Our findings highlight the necessity of robust and faithful unlearning methods for safe deployment of language models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 31 of its 36 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

31 of 36 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge26 Feb
  2. 2024Do Unlearning Methods Remove Information from Language Model Weights?11 Oct
  3. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  4. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning19 Jun
  5. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models16 Jun
  6. 2024Adversarial Machine Unlearning11 Jun
  7. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models6 May
  8. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  9. 2024Machine Unlearning of Pre-trained Large Language Models23 Feb
  10. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  11. 2023Large Language Model Unlearning14 Oct
  12. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
1 more
  1. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  12. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
6 more
  1. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  4. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  5. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
  6. 2025Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraintscited by 0
  2. 2026WIN-U: Woodbury-Informed Newton-Unlearning as a retain-free Machine Unlearning Frameworkcited by 0
  3. 2026Hidden not Deleted: How Networks Suppress Entangled Featurescited by 0
  4. 2026Faithful Dual-constrained Erasure for Robust LLM Safety Alignmentcited by 0
  5. 2026Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealingcited by 0