mlsanityData to 7 Oct 2026

Realistic Evaluation of Model Merging for Compositional Generalization

arXiv 2409.18314 ↗First posted 26 Sep 2024cs.LG cs.CL cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Merging has become a widespread way to cheaply combine individual models into a single model that inherits their capabilities and attains better performance. This popularity has spurred rapid development of many new merging methods, which are typically validated in disparate experimental settings and frequently differ in the assumptions made about model architecture, data availability, and computational budget. In this work, we characterize the relative merits of different merging methods by evaluating them in a shared experimental setting and precisely identifying the practical requirements of each method. Specifically, our setting focuses on using merging for compositional generalization of capabilities in image classification, image generation, and natural language processing. Additionally, we measure the computational costs of different merging methods as well as how they perform when scaling the number of models being merged. Taken together, our results clarify the state of the field of model merging and provide a comprehensive and rigorous experimental setup to test new methods.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 58 of its 71 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

58 of 71 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  13. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  14. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  15. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  16. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  17. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  18. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  19. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  20. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  21. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  22. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  23. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  24. 2019Model Fusion via Optimal Transportcited by 217
  25. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  26. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  27. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  28. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  29. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  30. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  31. 2022Fusing finetuned models for better pretrainingcited by 110
  32. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  33. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  34. 2024Multi-LoRA Composition for Image Generationcited by 80
  35. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  36. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  37. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  38. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  2. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  3. 2025Subspace-Boosted Model Mergingcited by 9
  4. 2025Soup-of-Experts: Pretraining Specialist Models via Parameters Averagingcited by 3
  5. 2025Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacycited by 3
  6. 2025Learning Composable Chains-of-Thoughtcited by 3
  7. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  8. 2026Bridging Domains through Subspace-Aware Model Mergingcited by 2
  9. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  10. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  11. 2025Efficient Multi-Source Knowledge Transfer by Model Mergingcited by 0
  12. 2026Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Expertscited by 0