mlsanityData to 7 Oct 2026

ZipIt! Merging Models from Different Tasks without Training

arXiv 2305.03053 ↗First posted 4 May 2023cs.CV cs.LGcited by 178

Not on the map. Why?

Abstract (arXiv)

Typical deep visual recognition models are capable of performing the one task they were trained on. In this paper, we tackle the extremely difficult problem of combining distinct models with different initializations, each solving a separate task, into one multi-task model without any additional training. Prior work in model merging permutes one model to the space of the other then averages them together. While this works for models trained on the same task, we find that this fails to account for the differences in models trained on disjoint tasks. Thus, we introduce "ZipIt!", a general method for merging two arbitrary models of the same architecture that incorporates two simple strategies. First, in order to account for features that aren't shared between models, we expand the model merging problem to allow for merging features within each model by defining a general "zip" operation. Second, we add support for partially zipping the models up until a specified layer, naturally creating a multi-head model. We find that these two changes combined account for 20-60% improvement over prior work, making it more feasible to merge models trained on disjoint tasks without retraining.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 56 of its 77 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

56 of 77 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  9. 2017Mask R-CNNcited by 7,259
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2016Learning without Forgettingcited by 2,465
  3. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2022Token Merging: Your ViT But Fastercited by 995
  10. 2021Scaling Vision Transformerscited by 993
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  13. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  16. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  17. 2013Domain Generalization via Invariant Feature Representationcited by 622
  18. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  19. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  20. 2020Federated Learning with Matched Averagingcited by 564
  21. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  22. 2019YOLACT: Real-time Instance Segmentationcited by 441
  23. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  24. 2020What is being transferred in transfer learning?cited by 365
  25. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  26. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  27. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  28. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  29. 2019Model Fusion via Optimal Transportcited by 217
  30. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  31. 2022Patching open-vocabulary models by interpolating weightscited by 187
  32. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  33. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  34. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  35. 2022Fusing finetuned models for better pretrainingcited by 110
  36. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  37. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  38. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 59; 2025: 58; 2026: 40.

178 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 362
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  5. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  6. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  7. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  8. 2024Model merging with SVD to tie the Knotscited by 118
  9. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  10. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  11. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
38 more
  1. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  4. 2024Parameter Competition Balancing for Model Mergingcited by 71
  5. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  6. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  7. 2023Learning from models beyond fine-tuningcited by 47
  8. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  9. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  10. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  11. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  12. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 36
  13. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  14. 2023Transformer Fusion with Optimal Transportcited by 35
  15. 2023Equivariant Deep Weight Space Alignmentcited by 34
  16. 2024Training-Free Pretrained Model Mergingcited by 32
  17. 2023Task Arithmetic with LoRA for Continual Learningcited by 31
  18. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 31
  19. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  20. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  21. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  22. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 24
  23. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  24. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  25. 2024C2M3: Cycle-Consistent Multi-Model Mergingcited by 20
  26. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  27. 2024StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elementscited by 18
  28. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 18
  29. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  30. 2023Merging Vision Transformers from Different Tasks and Domainscited by 17
  31. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  32. 2025Update Your Transformer to the Latest Release: Re-Basin of Task Vectorscited by 17
  33. 2024Merging Text Transformer Models from Different Initializationscited by 16
  34. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  35. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  36. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 14
  37. 2024Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluationcited by 14
  38. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14

50 most cited of 178 All 178 (JSON)