mlsanityData to 14 Oct 2026

Per-parameter Task Arithmetic for Unlearning in Large Language Models

arXiv 2601.22030 ↗First posted 29 Jan 2026cs.LGcited by 2

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the parameter difference between a privacy-information-tuned model and the original model. While efficient, it can cause over-forgetting by disrupting parameters essential for retaining other information. Motivated by the observation that each parameter exhibits different importance for forgetting versus retention, we propose a per-parameter task arithmetic (PerTA) mechanism to rescale the TV, allowing per-parameter adjustment. These weights quantify the relative importance of each parameter for forgetting versus retention, estimated via gradients (i.e., PerTA-grad) or the diagonal Fisher information approximation (i.e., PerTA-fisher). Moreover, we discuss the effectiveness of PerTA, extend it to a more general form, and provide further analysis. Extensive experiments demonstrate that PerTA consistently improves upon standard TV, and in many cases surpasses widely used training-based unlearning methods in both forgetting effectiveness and overall model utility. By retaining the efficiency of task arithmetic while mitigating over-forgetting, PerTA offers a principled and practical framework for LLM unlearning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 50 of its 57 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

50 of 57 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  2. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality8 Jun
  3. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning17 May
  4. 2025GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs12 Mar
  5. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  6. 2025LLM Unlearning via Neural Activation Redirection11 Feb
  7. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
  8. 2024Catastrophic Failure of LLM Unlearning via Quantization21 Oct
  9. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  10. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  11. 2024Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs17 Jun
  12. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods13 Jun
10 more
  1. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  3. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  5. 2023Large Language Model Unlearning14 Oct
  6. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
  9. 2022Continual Learning and Private Unlearning24 Mar
  10. 2021Fast Yet Effective Machine Unlearning17 Nov

Other references · 28

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2019Machine Unlearningcited by 1,061
  7. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  8. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  9. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  10. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  11. 2014New insights and perspectives on the natural gradient methodcited by 443
  12. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
16 more
  1. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  2. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  3. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  5. 2024Knowledge Fusion of Large Language Modelscited by 115
  6. 2023Copyright Violations and Large Language Modelscited by 108
  7. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  8. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  9. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  10. 2023Beyond Size: How Gradients Shape Pruning Decisions in Large Language Modelscited by 30
  11. 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
  12. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  13. 2024Model Composition for Multimodal Large Language Modelscited by 12
  14. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  15. 2024SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Modelscited by 11
  16. 2024Knowledge Fusion of Chat LLMs: A Preliminary Technical Reportcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vectorcited by 0
  2. 2026UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs
  2. 2025iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss
  3. 2024Machine Unlearning of Pre-trained Large Language Models
  4. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
  5. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methods
  6. 2026Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning
  7. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
  8. 2025Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning