mlsanityData to 7 Oct 2026

Understanding Machine Unlearning Through the Lens of Mode Connectivity

arXiv 2504.06407 ↗First posted 8 Apr 2025cs.LG cs.AI cs.CLcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood. In this paper, we study machine unlearning through the lens of mode connectivity--the phenomenon that independently trained models can often be connected by smooth low-loss paths in parameter space. We introduce em mode connectivity in unlearning (MCU) and evaluate it across a range of settings, including curriculum learning, second-order optimization, and connectivity across different unlearning methods. We find that many unlearned models lie in connected basins with smooth retain/forget behavior, while changes in training dynamics can move solutions into different basins. MCU also reveals that models within the same basin can differ substantially on privacy metrics, and that unlearning progresses nonlinearly from the original model to the unlearned model. In addition, linear connectivity suggests that most approximate unlearning methods are mechanistically distinct from retraining. Finally, MCU-based ensembling can improve generalization and robustness to relearning attacks, and MCU smoothness correlates with unlearning difficulty. To our knowledge, this is the first study of machine unlearning through the lens of mode connectivity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 53 of its 69 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

53 of 69 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
  2. 2025Tool Unlearning for Tool-Augmented LLMs3 Feb
  3. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct
  4. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  5. 2024Dissecting Fine-Tuning Unlearning in Large Language Models9 Oct
  6. 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
  7. 2024Towards Certified Unlearning for Deep Neural Networks1 Aug
  8. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  9. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning21 Jun
  10. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning19 Jun
  11. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  12. 2024What makes unlearning hard and what to do about it3 Jun
14 more
  1. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models6 May
  2. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  3. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  4. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning12 Mar
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  6. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  7. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection7 Dec
  8. 2023MultiDelete for Multimodal Machine Unlearning18 Nov
  9. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  10. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  11. 2023Model Sparsity Can Simplify Machine Unlearning11 Apr
  12. 2022Adversarial Unlearning: Reducing Confidence Along Adversarial Directions3 Jun
  13. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
  14. 2021Machine Unlearning via Algorithmic Stability25 Feb

Other references · 27

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2019Machine Unlearningcited by 1,020
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  9. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2020Amnesiac Machine Learningcited by 305
15 more
  1. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  3. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 143
  4. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  5. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  6. 2022Mechanistic Mode Connectivitycited by 50
  7. 2023GNNDelete: A General Strategy for Unlearning in Graph Neural Networkscited by 43
  8. 2023Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examplescited by 40
  9. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  10. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  11. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  12. 2024Tilted Sharpness-Aware Minimizationcited by 6
  13. 2025Linguistic Blind Spots of Large Language Modelscited by 6
  14. 2024Input Space Mode Connectivity in Deep Neural Networkscited by 4
  15. 2025FROG: Fair Removal on Graphscited by 4

Built on by

Not cited yet.