# Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics

arXiv: https://arxiv.org/abs/2411.16139 · HTML: https://mlsanity.com/paper/2411.16139/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [Parameter Competition Balancing for Model Merging](https://mlsanity.com/paper/2410.02396/) (2024)
- [Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model](https://mlsanity.com/paper/2401.17868/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning](https://mlsanity.com/paper/2212.01378/) (2022)
- [A Gradient Flow Framework For Analyzing Network Pruning](https://mlsanity.com/paper/2009.11839/) (2020)
- [Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective](https://mlsanity.com/paper/2310.11451/) (2023)
- [Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs](https://mlsanity.com/paper/2310.02277/) (2023)

## Built on by (most cited first, 11 of 11)

- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts](https://mlsanity.com/paper/2501.15065/) (2025)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge](https://mlsanity.com/paper/2502.20186/) (2025)
- [Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging](https://mlsanity.com/paper/2502.12217/) (2025)
- [AnyEdit++: Adaptive Long-Form Knowledge Editing via Bayesian Surprise](https://mlsanity.com/paper/2606.01053/) (2026)
- [You Only Train Once: A Flexible Training Framework for Code Vulnerability Detection Driven by Vul-Vector](https://mlsanity.com/paper/2506.10988/) (2025)
- [Look the Other Way: Designing 'Positive' Molecules with Negative Data via Task Arithmetic](https://mlsanity.com/paper/2507.17876/) (2025)
- [Saliency-Aware Model Merging](https://mlsanity.com/paper/2606.00511/) (2026)
- [CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation](https://mlsanity.com/paper/2608.12842/) (2026)

## Abstract (arXiv)

Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these models in real-world multi-task learning (MTL) scenarios poses substantial challenges, primarily due to high computational costs and inefficiencies in inference. Traditional approaches such as pruning, quantization, and knowledge distillation have been explored to mitigate these issues, but they often fall short in fully addressing the complexities of multi-task environments. This paper introduces textbfunderlineSelective textbfunderlineTask textbfunderlineArithmetic underline(STA), a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. STA addresses three key challenges: (i) Parameter importance diversity:  Recognizing that different tasks relie on distinct parameters, STA employs a loss-sensitive parameter importance metric derived from a first-order Taylor expansion to accurately measure the importance of parameters for each task. (ii) Over-reliance on hyperparameter tuning: By enhancing the sparsity of task vectors through parameter importance metrics, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model. (iii) Neglect of other abilities in task arithmetic:  Previous works have largely overlooked the potential for more precise task forgetting. STA leverages its parameter importance metric to achieve more controlled and effective task forgetting, minimizing the impact of noisy elements that can degrade model performance. Experimental results demonstrate that STA achieves superior multi-task performance across benchmarks and excellent performance in task forgetting.