mlsanityData to 14 Oct 2026

Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

arXiv 2305.12827 ↗First posted 22 May 2023cs.LG cs.CVcited by 221

Not on the map. Why?

Abstract (arXiv)

Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space: By adding the fine-tuned weights of different tasks, the model's performance can be improved on these tasks, while negating them leads to task forgetting. Yet, our understanding of the effectiveness of task arithmetic and its underlying principles remains limited. We present a comprehensive study of task arithmetic in vision-language models and show that weight disentanglement is the crucial factor that makes it effective. This property arises during pre-training and manifests when distinct directions in weight space govern separate, localized regions in function space associated with the tasks. Notably, we show that fine-tuning models in their tangent space by linearizing them amplifies weight disentanglement. This leads to substantial performance improvements across multiple task arithmetic benchmarks and diverse models. Building on these findings, we provide theoretical and empirical analyses of the neural tangent kernel (NTK) of these models and establish a compelling link between task arithmetic and the spatial localization of the NTK eigenfunctions. Overall, our work uncovers novel insights into the fundamental mechanisms of task arithmetic and offers a more reliable and effective approach to edit pre-trained models through the NTK linearization.

Key facts. Cited by 221 papers in the map (data through 14 Oct 2026). Builds on 69 of its 93 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

69 of 93 references.

References · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
38 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  4. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  7. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  8. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  9. 2015Automatic differentiation in machine learning: a surveycited by 855
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  13. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  14. 2018On Lazy Training in Differentiable Programmingcited by 675
  15. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  16. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  17. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  18. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  19. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  20. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  21. 2018Towards a Definition of Disentangled Representationscited by 296
  22. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  23. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  24. 2020Supermasks in Superpositioncited by 227
  25. 2019Model Fusion via Optimal Transportcited by 219
  26. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  27. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  28. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  29. 2022Patching open-vocabulary models by interpolating weightscited by 190
  30. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  31. 2020Training independent subnetworks for robust predictioncited by 177
  32. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  33. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  34. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  35. 2022Fusing finetuned models for better pretrainingcited by 110
  36. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  37. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  38. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 61; 2025: 80; 2026: 67.

221 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  4. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  5. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  6. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  7. 2024Model merging with SVD to tie the Knotscited by 122
  8. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  9. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  10. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  11. 2023Deep Model Fusion: A Surveycited by 91
  12. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
38 more
  1. 2024Parameter Competition Balancing for Model Mergingcited by 72
  2. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  3. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65
  4. 2024What Matters for Model Merging at Scale?cited by 55
  5. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  6. 2024On Large Language Model Continual Unlearningcited by 52
  7. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  8. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  9. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 45
  10. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  11. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  12. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  13. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  14. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  15. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  16. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  17. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 33
  18. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 29
  19. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  20. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  21. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 26
  22. 2024Revisiting Weight Averaging for Model Mergingcited by 25
  23. 2025The Universal Weight Subspace Hypothesiscited by 24
  24. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  25. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 23
  26. 2023Layer-wise Linear Mode Connectivitycited by 22
  27. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  28. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  29. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21
  30. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  31. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 20
  32. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  33. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 19
  34. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  35. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  36. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 18
  37. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  38. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18

50 most cited of 221 All 221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Steering Language Models with Weight Arithmetic
  2. 2026First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
  3. 2025From Memorization to Reasoning in the Spectrum of Loss Curvature
  4. 2026ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models
  5. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexity
  6. 2026Training-Free Task Vectors for LLM Behavioral Control
  7. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  8. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure