mlsanityData to 7 Oct 2026

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

arXiv 2408.07666 ↗First posted 14 Aug 2024cs.LG cs.AI cs.CLcited by 247

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and more than ten machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). Builds on 206 of its 297 references that are in the map. First posted on arXiv 14 Aug 2024.

Builds on

206 of 297 references.

References · 206

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2024Mixtral of Expertscited by 1,785
  6. 2023Consistency Modelscited by 1,778
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  13. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  14. 2022Editing Models with Task Arithmeticcited by 1,076
  15. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  16. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  17. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  18. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  19. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  20. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  21. 2021Robust fine-tuning of zero-shot modelscited by 785
  22. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  23. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  24. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  25. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  26. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  27. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  28. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  29. 2020Federated Learning with Matched Averagingcited by 564
  30. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  31. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  32. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  33. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  34. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  35. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  36. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  37. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  38. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359

50 most cited of 206 All 206 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 117; 2026: 107.

247 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  3. 2025Efficient Reasoning Models: A Surveycited by 93
  4. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  5. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  6. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  7. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  8. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  9. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  10. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  11. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  12. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 35
38 more
  1. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  2. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  3. 2024ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Modelscited by 26
  4. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  5. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  6. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  7. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  8. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  9. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  10. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  11. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  12. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  13. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  14. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  15. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  16. 2025Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captionscited by 13
  17. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  18. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 12
  19. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  20. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  21. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 12
  22. 2025STAR: Spectral Truncation and Rescale for Model Mergingcited by 11
  23. 2025Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracycited by 11
  24. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  25. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  26. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  27. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  28. 2024LoBAM: LoRA-Based Backdoor Attack on Model Mergingcited by 10
  29. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 10
  30. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  31. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  32. 2025Revisiting Model Interpolation for Efficient Reasoningcited by 10
  33. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  34. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 9
  35. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  36. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  37. 2025UCD: Unlearning in LLMs via Contrastive Decodingcited by 9
  38. 2025Subspace-Boosted Model Mergingcited by 9

50 most cited of 247 All 247 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  2. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
  3. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
  4. 2024Unconstrained Model Merging for Enhanced LLM Reasoning
  5. 2025Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
  6. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
  7. 2024MoD: A Distribution-Based Approach for Merging Large Language Models
  8. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization