mlsanityData to 14 Oct 2026

From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models

arXiv 2609.19553 ↗First posted 17 Sep 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Model fusion integrates the capabilities from source models into a single target model. As of June 2026, Hugging Face hosts more than 2M models. This growing pool provides a rich base for model reuse and capability integration. Yet existing surveys often cover only separate parts of this space, and they do not provide a unified definition or a systematic taxonomy. This survey defines model fusion and organizes prior work into three levels: parameter-level, representation-level, and behavior-level fusion. We also review related metrics, benchmarks, and applications, summarize current challenges, and identify future directions. Our goal is to provide a clear map of this area and support future work on model fusion. A comprehensive list of papers about model fusion is available at https://github.com/Baicaihaochi/Awesome-Model-Fusion-Survey.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 105 of its 130 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

105 of 130 references.

References · 105

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
38 more
  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  5. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  6. 2022Teaching Small Language Models to Reasoncited by 284
  7. 2026Self-Distillation Enables Continual Learningcited by 247
  8. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  10. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  11. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  12. 2024Model merging with SVD to tie the Knotscited by 122
  13. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  14. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  15. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  16. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  17. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  18. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 93
  19. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  20. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  21. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 60
  22. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  23. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
  24. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  25. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  26. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52
  27. 2023Learning from models beyond fine-tuningcited by 48
  28. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  29. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  30. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 43
  31. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  32. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  33. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 36
  34. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 33
  35. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  36. 2025Activation-Informed Merging of Large Language Modelscited by 30
  37. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 26
  38. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 26

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SMAT: Simple and Efficient Merge-Aware Trainingcited by 0