# Training-Free Pretrained Model Merging

arXiv: https://arxiv.org/abs/2403.01753 · HTML: https://mlsanity.com/paper/2403.01753/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 63 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [Composing Parameter-Efficient Modules with Arithmetic Operations](https://mlsanity.com/paper/2306.14870/) (2023)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)

## Built on by (most cited first, 30 of 32)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Parameter Competition Balancing for Model Merging](https://mlsanity.com/paper/2410.02396/) (2024)
- [Retraining-Free Merging of Sparse MoE via Hierarchical Clustering](https://mlsanity.com/paper/2410.08589/) (2024)
- [PLeaS -- Merging Models with Permutations and Least Squares](https://mlsanity.com/paper/2407.02447/) (2024)
- [Exact Unlearning of Finetuning Data via Model Merging at Scale](https://mlsanity.com/paper/2504.04626/) (2025)
- [SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery](https://mlsanity.com/paper/2410.14389/) (2024)
- [LoBAM: LoRA-Based Backdoor Attack on Model Merging](https://mlsanity.com/paper/2411.16746/) (2024)
- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild](https://mlsanity.com/paper/2410.05357/) (2024)
- [Training-free Heterogeneous Model Merging](https://mlsanity.com/paper/2501.00061/) (2024)
- [Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning](https://mlsanity.com/paper/2507.04790/) (2025)
- [Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs](https://mlsanity.com/paper/2504.10902/) (2025)
- [Label-Free Cross-Task LoRA Merging with Null-Space Compression](https://mlsanity.com/paper/2603.26317/) (2026)
- [Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering](https://mlsanity.com/paper/2410.04801/) (2024)
- [Exploring Model Kinship for Merging Large Language Models](https://mlsanity.com/paper/2410.12613/) (2024)
- [Training-Free, Identity-Preserving Image Editing for Fashion Pose Alignment and Normalization](https://mlsanity.com/paper/2501.13692/) (2025)
- [MAGIC: Achieving Superior Model Merging via Magnitude Calibration](https://mlsanity.com/paper/2512.19320/) (2025)
- [Bridging Domains through Subspace-Aware Model Merging](https://mlsanity.com/paper/2603.05768/) (2026)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [The Impact of Model Zoo Size and Composition on Weight Space Learning](https://mlsanity.com/paper/2504.10141/) (2025)
- [Merge-Friendly Post-Training Quantization for Multi-Target Domain Adaptation](https://mlsanity.com/paper/2505.23651/) (2025)
- [SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging](https://mlsanity.com/paper/2506.18135/) (2025)
- [StereoVGGT: A Training-Free Visual Geometry Transformer for Stereo Vision](https://mlsanity.com/paper/2603.29368/) (2026)
- [Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation](https://mlsanity.com/paper/2605.22350/) (2026)
- [Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Training](https://mlsanity.com/paper/2607.19971/) (2026)
- [MIN-Merging: Merge the Important Neurons for Model Merging](https://mlsanity.com/paper/2510.17890/) (2025)
- [MergeRec: Model Merging for Data-Isolated Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2601.01753/) (2026)
- [Training-Free Cross-Architecture Merging for Graph Neural Networks](https://mlsanity.com/paper/2602.19332/) (2026)
- [Domain-Adaptive Model Merging Across Disconnected Modes](https://mlsanity.com/paper/2603.05957/) (2026)
- [Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning](https://mlsanity.com/paper/2605.08311/) (2026)

## Abstract (arXiv)

Recently, model merging techniques have surfaced as a solution to combine multiple single-talent models into a single multi-talent model. However, previous endeavors in this field have either necessitated additional training or fine-tuning processes, or require that the models possess the same pre-trained initialization. In this work, we identify a common drawback in prior works w.r.t. the inconsistency of unit similarity in the weight space and the activation space. To address this inconsistency, we propose an innovative model merging framework, coined as merging under dual-space constraints (MuDSC). Specifically, instead of solely maximizing the objective of a single space, we advocate for the exploration of permutation matrices situated in a region with a unified high similarity in the dual space, achieved through the linear combination of activation and weight similarity matrices. In order to enhance usability, we have also incorporated adaptations for group structure, including Multi-Head Attention and Group Normalization. Comprehensive experimental comparisons demonstrate that MuDSC can significantly boost the performance of merged models with various task combinations and architectures. Furthermore, the visualization of the merged model within the multi-task loss landscape reveals that MuDSC enables the merged model to reside in the overlapping segment, featuring a unified lower loss for each task. Our code is publicly available at https://github.com/zju-vipa/training_free_model_merging.