mlsanityData to 7 Oct 2026

Model merging with SVD to tie the Knots

arXiv 2410.19735 ↗First posted 25 Oct 2024cs.CVcited by 118

Not on the map. Why?

Abstract (arXiv)

Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

45 of 57 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
33 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  8. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  13. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  14. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  15. 2020What is being transferred in transfer learning?cited by 365
  16. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  17. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  18. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  19. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  20. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  21. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  22. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  23. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  24. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  25. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  26. 2022Fusing finetuned models for better pretrainingcited by 110
  27. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  28. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  29. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  30. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  31. 2023Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?cited by 13
  32. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  33. 2021Towards Consistent Predictive Confidence through Fitted Ensemblescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 44; 2026: 72.

118 papers.

Sort:

By area:

  1. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  2. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  3. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  4. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  5. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  6. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  7. 2025The Universal Weight Subspace Hypothesiscited by 22
  8. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  9. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 18
  10. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  11. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  12. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
38 more
  1. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  2. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  3. 2025Subspace-Boosted Model Mergingcited by 9
  4. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  5. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  6. 2026Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learningcited by 8
  7. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  8. 2025Permissioned LLMs: Enforcing Access Control in Large Language Modelscited by 7
  9. 2025ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptationcited by 7
  10. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  11. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  12. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  13. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  14. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 6
  15. 2026Model Merging in the Essential Subspacecited by 6
  16. 2025Superpose Task-specific Features for Model Mergingcited by 5
  17. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  18. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 5
  19. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  20. 2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 4
  21. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  22. 2025ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptationcited by 3
  23. 2025K-Merge: Online Continual Merging of Adapters for On-device Large Language Modelscited by 3
  24. 2025CALM: A CKA-Guided Adaptive Layer-Wise Modularization Framework for LLM Quantizationcited by 3
  25. 2026Will it Merge? On The Causes of Model Mergeabilitycited by 3
  26. 2026Orthogonal Model Mergingcited by 3
  27. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  28. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 3
  29. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  30. 2025ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Mergingcited by 2
  31. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 2
  32. 2025Low-Rank Expert Merging for Multi-Source Domain Adaptation in Person Re-Identificationcited by 2
  33. 2025Purifying Task Vectors in Knowledge-Aware Subspace for Model Mergingcited by 2
  34. 2025Hierarchical Federated Unlearning for Large Language Modelscited by 2
  35. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  36. 2025Defending Unauthorized Model Merging via Dual-Stage Weight Protectioncited by 2
  37. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  38. 2026Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Mergingcited by 2

50 most cited of 118 All 118 (JSON)