mlsanityData to 7 Oct 2026

Task Singular Vectors: Reducing Task Interference in Model Merging

arXiv 2412.00081 ↗First posted 26 Nov 2024cs.LG stat.MLcited by 143

Not on the map. Why?

Abstract (arXiv)

Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

37 of 59 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
25 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  3. 2018Deep Learning for Classical Japanese Literaturecited by 415
  4. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  5. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  6. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  7. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  8. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  9. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  10. 2013Predicting Parameters in Deep Learningcited by 223
  11. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  12. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  13. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  14. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  15. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  16. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  17. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  18. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  19. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  20. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  21. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  22. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  23. 2024C2M3: Cycle-Consistent Multi-Model Mergingcited by 20
  24. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 10
  25. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 49; 2026: 92.

143 papers.

Sort:

By area:

  1. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  2. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  3. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  4. 2024Revisiting Weight Averaging for Model Mergingcited by 24
  5. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  6. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  7. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  8. 2025RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Mergingcited by 10
  9. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 9
  10. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  11. 2025Subspace-Boosted Model Mergingcited by 9
  12. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 8
38 more
  1. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  2. 2026Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learningcited by 8
  3. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  4. 2025Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMscited by 7
  5. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  6. 2025Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translationcited by 6
  7. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  8. 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
  9. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunkingcited by 6
  10. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 6
  11. 2026Model Merging in the Essential Subspacecited by 6
  12. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 5
  13. 2025Task Vector Quantization for Memory-Efficient Model Mergingcited by 5
  14. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  15. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 5
  16. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  17. 2025Mergenetic: a Simple Evolutionary Model Merging Librarycited by 4
  18. 2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 4
  19. 2025Model Merging with Functional Dual Anchorscited by 4
  20. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  21. 2026Transporting Task Vectors across Different Architectures without Trainingcited by 4
  22. 2025On Fairness of Task Arithmetic: The Role of Task Vectorscited by 3
  23. 2025Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Modelscited by 3
  24. 2025Towards Reversible Model Merging For Low-rank Weightscited by 3
  25. 2026Orthogonal Model Mergingcited by 3
  26. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  27. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 3
  28. 2026Understanding and Enforcing Weight Disentanglement in Task Arithmeticcited by 3
  29. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  30. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 2
  31. 2025Low-Rank Expert Merging for Multi-Source Domain Adaptation in Person Re-Identificationcited by 2
  32. 2025Purifying Task Vectors in Knowledge-Aware Subspace for Model Mergingcited by 2
  33. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  34. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  35. 2025MAGIC: Achieving Superior Model Merging via Magnitude Calibrationcited by 2
  36. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  37. 2026Bridging Domains through Subspace-Aware Model Mergingcited by 2
  38. 2026One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Mergingcited by 2

50 most cited of 143 All 143 (JSON)