mlsanityData to 7 Oct 2026

NegMerge: Sign-Consensual Weight Merging for Machine Unlearning

arXiv 2410.05583 ↗First posted 8 Oct 2024cs.LG cs.AIcited by 9

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning aims to selectively remove specific knowledge from a trained model. Existing approaches, such as Task Arithmetic, fine-tune the model on the forget set to create a task vector (i.e., a direction in weight space) for subtraction from the original model's weight. However, their effectiveness is highly sensitive to hyperparameter selection, requiring extensive validation to identify the optimal vector from many fine-tuned candidates. In this paper, we propose a novel method that utilizes all fine-tuned models trained with varying hyperparameters instead of a single selection. Specifically, we aggregate the computed task vectors by retaining only the elements with consistent shared signs. The merged task vector is then negated to induce unlearning on the original model. Evaluations on zero-shot and standard image recognition tasks across twelve datasets and four backbone architectures show that our approach outperforms state-of-the-art methods while requiring similar or fewer computational resources. Code is available at https://github.com/naver-ai/negmerge.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 38 of its 47 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

38 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  2. 2023Model Sparsity Can Simplify Machine Unlearning11 Apr
  3. 2023Boundary Unlearning21 Mar
  4. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep
  5. 2021Machine Unlearning of Features and Labels26 Aug

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
21 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2021Membership Inference Attacks From First Principlescited by 839
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  8. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  10. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  11. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  12. 2022Patching open-vocabulary models by interpolating weightscited by 187
  13. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  14. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  15. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  16. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  17. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  18. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  19. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  20. 2017NSML: A Machine Learning Platform That Enables You to Focus on Your Modelscited by 34
  21. 2023Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  2. 2025Geometric-disentangelment Unlearningcited by 4
  3. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  4. 2026Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Modelscited by 1
  5. 2026SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vectorcited by 0
  6. 2026HyperFix: Combinatorial Nonlinear Correction for Task Vector Mergingcited by 0
  7. 2026Open Vocabulary Domain Unlearningcited by 0
  8. 2026UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?cited by 0
  9. 2026Unmerge: Efficient Machine Unlearning via Task Arithmeticcited by 0