mlsanityData to 14 Oct 2026

Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging

arXiv 2406.15479 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 116

Not on the map. Why?

Abstract (arXiv)

In the era of large language models, model merging is a promising way to combine multiple task-specific models into a single multitask model without extra training. However, two challenges remain: (a) interference between different models and (b) heterogeneous data during testing. Traditional model merging methods often show significant performance gaps compared to fine-tuned models due to these issues. Additionally, a one-size-fits-all model lacks flexibility for diverse test data, leading to performance degradation. We show that both shared and exclusive task-specific knowledge are crucial for merging performance, but directly merging exclusive knowledge hinders overall performance. In view of this, we propose Twin-Merging, a method that encompasses two principal stages: (1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency; (2) dynamically merging shared and task-specific knowledge based on the input. This approach narrows the performance gap between merged and fine-tuned models and improves adaptability to heterogeneous data. Extensive experiments on 20 datasets for both language and vision tasks demonstrate the effectiveness of our method, showing an average improvement of 28.34% in absolute normalized score for discriminative tasks and even surpassing the fine-tuned upper bound on the generative tasks. Our implementation is available in urlhttps://github.com/LZY-the-boys/Twin-Merging

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 55 of its 75 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

55 of 75 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2013Describing Textures in the Wildcited by 2,125
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  6. 2022Editing Models with Task Arithmeticcited by 1,120
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  8. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  11. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  13. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  14. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  15. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  16. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  17. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  18. 2020Federated Learning with Matched Averagingcited by 564
  19. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  20. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  21. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  22. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  23. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  24. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  25. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  26. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  27. 2019Model Fusion via Optimal Transportcited by 219
  28. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  29. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  30. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  31. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  32. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  33. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  34. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65
  35. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  36. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  37. 2023Fusing Models with Complementary Expertisecited by 46
  38. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 56; 2026: 41.

116 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  4. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  5. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 99
  6. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  7. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  8. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  9. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  10. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  11. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  12. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 26
38 more
  1. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  2. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  3. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  4. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 23
  5. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 23
  6. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  7. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  8. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 19
  9. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 18
  10. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  11. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  12. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  13. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  14. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 13
  15. 2025Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integrationcited by 13
  16. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  17. 2025Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Thinkcited by 12
  18. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  19. 2025Model Merging Scaling Laws in Large Language Modelscited by 12
  20. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11
  21. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  22. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  23. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10
  24. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 9
  25. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 9
  26. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  27. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  28. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  29. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  30. 2025LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Mergingcited by 7
  31. 2025On Task Vectors and Gradientscited by 7
  32. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 6
  33. 2025To See a World in a Spark of Neuron: Disentangling Multi-task Interference for Training-free Model Mergingcited by 6
  34. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  35. 2026MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Mergingcited by 6
  36. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 6
  37. 20251bit-Merging: Dynamic Quantized Merging for Large Language Modelscited by 5
  38. 2025CAMEx: Curvature-aware Merging of Expertscited by 5

50 most cited of 116 All 116 (JSON)