mlsanityData to 14 Oct 2026

EMR-Merging: Tuning-Free High-Performance Model Merging

arXiv 2405.17461 ↗First posted 23 May 2024cs.LG cs.CVcited by 110

Not on the map. Why?

Abstract (arXiv)

The success of pretrain-finetune paradigm brings about the release of numerous model weights. In this case, merging models finetuned on different tasks to enable a single model with multi-task capabilities is gaining increasing attention for its practicability. Existing model merging methods usually suffer from (1) significant performance degradation or (2) requiring tuning by additional data or training. In this paper, we rethink and analyze the existing model merging paradigm. We discover that using a single model's weights can hardly simulate all the models' performance. To tackle this issue, we propose Elect, Mask & Rescale-Merging (EMR-Merging). We first (a) elect a unified model from all the model weights and then (b) generate extremely lightweight task-specific modulators, including masks and rescalers, to align the direction and magnitude between the unified model and each specific model, respectively. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance. We find that EMR-Merging shows outstanding performance compared to existing merging methods under different classical and newly-established settings, including merging different numbers of vision models (up to 30), NLP models, PEFT models, and multi-modal models.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Builds on 50 of its 91 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

50 of 91 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  13. 2018Neural Network Acceptability Judgmentscited by 958
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  15. 2017A Survey on Multi-Task Learningcited by 840
  16. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  17. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  18. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  19. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  20. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  21. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  22. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  23. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  24. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  25. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  26. 2021Vision Transformers are Robust Learnerscited by 207
  27. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  28. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  29. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  30. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  31. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 64
  32. 2023Uni3D: A Unified Baseline for Multi-dataset 3D Object Detectioncited by 39
  33. 2017Fruit recognition from images using deep learningcited by 35
  34. 2023Merging Vision Transformers from Different Tasks and Domainscited by 17
  35. 2022MangoLeafBD: A Comprehensive Image Dataset to Classify Diseased and Healthy Mango Leavescited by 13
  36. 2022Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafingcited by 10
  37. 2019Scraping Social Media Photos Posted in Kenya and Elsewhere to Detect and Analyze Food Typescited by 6
  38. 2023Stimulative Training++: Go Beyond The Performance Limits of Residual Networkscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 53; 2026: 42.

110 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  3. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  4. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  5. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  6. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  7. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 26
  8. 2024Revisiting Weight Averaging for Model Mergingcited by 25
  9. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  10. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 19
  11. 2025Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learningcited by 19
  12. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 18
38 more
  1. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  2. 2025Personality Vector: Modulating Personality of Large Language Models by Model Mergingcited by 17
  3. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  4. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  5. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
  6. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  7. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 13
  8. 2025Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integrationcited by 13
  9. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  10. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11
  11. 2025STAR: Spectral Truncation and Rescale for Model Mergingcited by 11
  12. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  13. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  14. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10
  15. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 9
  16. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  17. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  18. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  19. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  20. 2026Model Merging in the Essential Subspacecited by 7
  21. 2025To See a World in a Spark of Neuron: Disentangling Multi-task Interference for Training-free Model Mergingcited by 6
  22. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  23. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 6
  24. 2025Task Vector Quantization for Memory-Efficient Model Mergingcited by 5
  25. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  26. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 5
  27. 2025Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planningcited by 5
  28. 2025Rethinking Layer-wise Model Merging through Chain of Mergescited by 5
  29. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  30. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  31. 2025FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognitioncited by 4
  32. 2025Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmeticcited by 4
  33. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  34. 2026SimMerge: Learning to Select Merge Operators from Similarity Signalscited by 4
  35. 2026An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapsecited by 4
  36. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  37. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  38. 2025Towards Reversible Model Merging For Low-rank Weightscited by 3

50 most cited of 110 All 110 (JSON)