mlsanityData to 7 Oct 2026

Attention Smoothing Is All You Need For Unlearning

arXiv 2603.01285 ↗First posted 1 Mar 2026cs.LG cs.AI cs.CLcited by 6

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 89 of its 108 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

89 of 108 references.

Closest ancestors · 47

Same direction, newest first

  1. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  2. 2025Not All Tokens Are Meant to Be Forgotten3 Jun
  3. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  4. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
  5. 2024Unified Parameter-Efficient Unlearning for LLMs30 Nov
  6. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization16 Oct
  7. 2024A Closer Look at Machine Unlearning for Large Language Models10 Oct
  8. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  9. 2024Mitigating Memorization In Language Models3 Oct
  10. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models20 Sep
  11. 2024On Effects of Steering Latent Representation for Large Language Model Unlearning12 Aug
  12. 2024On Large Language Model Continual Unlearning14 Jul
35 more
  1. 2024Learning to Refuse: Towards Mitigating Privacy Risks in LLMs14 Jul
  2. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  3. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models2 Jul
  4. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning19 Jun
  5. 2024Textual Unlearning Gives a False Sense of Unlearning19 Jun
  6. 2024Soft Prompting for Unlearning in Large Language Models17 Jun
  7. 2024Avoiding Copyright Infringement via Large Language Model Unlearning16 Jun
  8. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models16 Jun
  9. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods13 Jun
  10. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  11. 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
  12. 2024RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models4 Jun
  13. 2024What makes unlearning hard and what to do about it3 Jun
  14. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  15. 2024Offset Unlearning for Large Language Models17 Apr
  16. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  17. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methods2 Apr
  18. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Models13 Mar
  19. 2024Guardrail Baselines for Unlearning in LLMs5 Mar
  20. 2024Machine Unlearning of Pre-trained Large Language Models23 Feb
  21. 2024Towards Safer Large Language Models through Machine Unlearning15 Feb
  22. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models15 Feb
  23. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  24. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models8 Feb
  25. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  26. 2023Large Language Model Unlearning14 Oct
  27. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  28. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  29. 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening15 Aug
  30. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions8 Jul
  31. 2023Model Sparsity Can Simplify Machine Unlearning11 Apr
  32. 2023Towards Unbounded Machine Unlearning20 Feb
  33. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  34. 2021Machine Unlearning of Features and Labels26 Aug
  35. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 42

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2019Machine Unlearningcited by 1,020
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
30 more
  1. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  2. 2023Erasing Concepts from Diffusion Modelscited by 530
  3. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  7. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  8. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  9. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  10. 2020Query-Key Normalization for Transformerscited by 266
  11. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  12. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  13. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  14. 2021Alignment of Language Agentscited by 167
  15. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  16. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  17. 2021Graph Unlearningcited by 129
  18. 2021Federated Unlearning via Class-Discriminative Pruningcited by 108
  19. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  20. 2022Federated Unlearning: How to Efficiently Erase a Client in FL?cited by 90
  21. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  22. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  23. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  24. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  25. 2023Knowledge Sanitization of Large Language Modelscited by 34
  26. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  27. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  28. 2021Attention Temperature Matters in Abstractive Summarization Distillationcited by 14
  29. 2023tasksource: A Dataset Harmonization Framework for Streamlined NLP Multi-Task Learning and Evaluationcited by 11
  30. 2025Automatic Calibration for Membership Inference Attack on Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repaircited by 2
  2. 2026SoK: Unlearnability and Unlearning for Model Dememorizationcited by 0
  3. 2026Trajectory-Guided Forget-Recover Network for Continual LLM Unlearningcited by 0
  4. 2026ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Controlcited by 0
  5. 2026Evaluating Machine Unlearning in ASRcited by 0
  6. 2026UNBIND: UNlearning By INference-time Directional Steering for Code LLMscited by 0