mlsanityData to 14 Oct 2026

Training-free LLM Merging for Multi-task Learning

arXiv 2506.12379 ↗First posted 14 Jun 2025cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing (NLP) tasks. The release of open-source LLMs like LLaMA and Qwen has triggered the development of numerous fine-tuned models tailored for various tasks and languages. In this paper, we explore an important question: is it possible to combine these specialized models to create a unified model with multi-task capabilities. We introduces Hierarchical Iterative Merging (Hi-Merging), a training-free method for unifying different specialized LLMs into a single model. Specifically, Hi-Merging employs model-wise and layer-wise pruning and scaling, guided by contribution analysis, to mitigate parameter conflicts. Extensive experiments on multiple-choice and question-answering tasks in both Chinese and English validate Hi-Merging's ability for multi-task learning. The results demonstrate that Hi-Merging consistently outperforms existing merging techniques and surpasses the performance of models fine-tuned on combined datasets in most scenarios. Code is available at: https://github.com/Applied-Machine-Learning-Lab/Hi-Merging.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

26 of 43 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  10. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  11. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  12. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
14 more
  1. 2019Model Fusion via Optimal Transportcited by 219
  2. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  3. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  4. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  5. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  6. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  7. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  8. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 59
  9. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  10. 2024M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Frameworkcited by 32
  11. 2023Multi-Task Deep Recommender Systems: A Surveycited by 30
  12. 2024Large Language Model Distilling Medication Recommendation Modelcited by 26
  13. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21
  14. 2024Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decodingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Distilling Time Series Foundation Models for Efficient Forecastingcited by 11
  2. 2026Unlocking the Potential of Continual Model Merging: An ODE Perspectivecited by 1
  3. 2026A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Lawscited by 0
  4. 2026RMPL: Relation-aware Multi-task Progressive Learning with Stage-wise Training for Multimedia Event Extractioncited by 0
  5. 2026Hierarchical Wasserstein Merging for Multi-Domain Multi-Task Learning: From Specialists to a Generalistcited by 0
  6. 2026Predicting and Repairing Merge Collapse in Large Language Modelscited by 0