# On Task Vectors and Gradients

arXiv: https://arxiv.org/abs/2508.16082 · HTML: https://mlsanity.com/paper/2508.16082/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 32 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging](https://mlsanity.com/paper/2406.15479/) (2024)
- [Fusing finetuned models for better pretraining](https://mlsanity.com/paper/2204.03044/) (2022)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [Model Merging by Uncertainty-Based Gradient Matching](https://mlsanity.com/paper/2310.12808/) (2023)
- [Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs](https://mlsanity.com/paper/2406.16797/) (2024)
- [PopulAtion Parameter Averaging (PAPA)](https://mlsanity.com/paper/2304.03094/) (2023)
- [C2M3: Cycle-Consistent Multi-Model Merging](https://mlsanity.com/paper/2405.17897/) (2024)
- [Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD](https://mlsanity.com/paper/2204.12446/) (2022)

## Built on by (most cited first, 7 of 7)

- [Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success](https://mlsanity.com/paper/2601.22285/) (2026)
- [Defending Unauthorized Model Merging via Dual-Stage Weight Protection](https://mlsanity.com/paper/2511.11851/) (2025)
- [Zero-Shot Quantization via Weight-Space Arithmetic](https://mlsanity.com/paper/2604.03420/) (2026)
- [Can Model Merging Improve Aggregation in DiLoCo?](https://mlsanity.com/paper/2607.03011/) (2026)
- [Beyond Fixed Rounds: Data-Free Early Stopping for Practical Federated Learning](https://mlsanity.com/paper/2601.22669/) (2026)
- [Efficient Decentralized Multi-task Dataset Valuation via Model Merging](https://mlsanity.com/paper/2607.03346/) (2026)
- [On Emergent Capabilities and Model Merging](https://mlsanity.com/paper/2609.24504/) (2026)

## Abstract (arXiv)

Task arithmetic has emerged as a simple yet powerful technique for model merging, enabling the combination of multiple finetuned models into one. Despite its empirical success, a clear theoretical explanation of why and when it works is lacking. This paper provides a rigorous theoretical foundation for task arithmetic by establishing a connection between task vectors and gradients of the task losses. We show that under standard gradient descent, a task vector generated from one epoch of finetuning is exactly equivalent to the negative gradient of the loss, scaled by the learning rate. For the practical multi-epoch setting, we prove that this equivalence holds approximately, with a second-order error term that we explicitly bound for feed-forward networks. Our empirical analysis across seven vision benchmarks corroborates our theory, demonstrating that the first-epoch gradient dominates the finetuning trajectory in both norm and direction. A key implication is that merging models finetuned for only a single epoch often yields performance comparable to merging fully converged models. These findings reframe task arithmetic as a form of approximate multitask learning, providing a clear rationale for its effectiveness and highlighting the critical role of early training dynamics in model merging.