mlsanityData to 7 Oct 2026

Dataless Knowledge Fusion by Merging Weights of Language Models

arXiv 2212.09849 ↗First posted 19 Dec 2022cs.CL cs.LGcited by 384

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns. This creates a barrier to fusing knowledge across individual models to yield a better single model. In this paper, we study the problem of merging individual models built on different training data sets to obtain a single model that performs well both across all data set domains and can generalize on out-of-domain data. We propose a dataless knowledge fusion method that merges models in their parameter space, guided by weights that minimize prediction differences between the merged model and the individual models. Over a battery of evaluation settings, we show that the proposed method significantly outperforms baselines such as Fisher-weighted averaging or model ensembling. Further, we find that our method is a promising alternative to multi-task learning that can preserve or sometimes improve over the individual models without access to the training data. Finally, model merging is more efficient than training a multi-task model, thus making it applicable to a wider set of scenarios.

Key facts. Cited by 384 papers in the map (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

38 of 58 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
26 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  9. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  10. 2020What is being transferred in transfer learning?cited by 365
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2019Model Fusion via Optimal Transportcited by 217
  13. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  14. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  15. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  16. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  17. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  18. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  19. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 119
  20. 2022Fusing finetuned models for better pretrainingcited by 110
  21. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  22. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  23. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  24. 2017WASSA-2017 Shared Task on Emotion Intensitycited by 37
  25. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  26. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 117; 2025: 145; 2026: 93.

384 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  3. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  4. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  5. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  6. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  7. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  8. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  9. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  10. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  11. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  12. 2024Model merging with SVD to tie the Knotscited by 118
38 more
  1. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  2. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  3. 2024Knowledge Fusion of Large Language Modelscited by 110
  4. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  6. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  7. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  8. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  9. 2023Deep Model Fusion: A Surveycited by 89
  10. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  11. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  12. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  13. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  14. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  15. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  16. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  17. 2024Parameter Competition Balancing for Model Mergingcited by 71
  18. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  19. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  20. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  21. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  22. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  23. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  24. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  25. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  26. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  27. 2024What Matters for Model Merging at Scale?cited by 52
  28. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  29. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  30. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  31. 2023Learning from models beyond fine-tuningcited by 47
  32. 2024SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teachercited by 45
  33. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  34. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  35. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  36. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  37. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  38. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39

50 most cited of 384 All 384 (JSON)