mlsanityData to 7 Oct 2026

Fusing finetuned models for better pretraining

arXiv 2204.03044 ↗First posted 6 Apr 2022cs.CL cs.CV cs.LGcited by 110

Not on the map. Why?

Abstract (arXiv)

Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavour that few can undertake. In this paper, we create better base models at hardly any cost, by fusing multiple existing fine tuned models into one. Specifically, we fuse by averaging the weights of these models. We show that the fused model results surpass the pretrained model ones. We also show that fusing is often better than intertraining. We find that fusing is less dependent on the target task. Furthermore, weight decay nullifies intertraining effects but not those of fusing.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

41 of 68 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2020Meta-Learning in Neural Networks: A Surveycited by 944
29 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  6. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  7. 2020What is being transferred in transfer learning?cited by 365
  8. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  9. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  10. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  12. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  13. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  14. 2014Qualitatively characterizing neural network optimization problemscited by 186
  15. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  16. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  17. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  18. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  19. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  20. 2021Making Transformers Solve Compositional Taskscited by 72
  21. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  22. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 55
  23. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  24. 2017Emotion Intensities in Tweetscited by 32
  25. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  26. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  27. 2020Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"cited by 22
  28. 2021Rethinking Why Intermediate-Task Fine-Tuning Workscited by 18
  29. 2021On Neurons Invariant to Sentence Structural Changes in Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 26; 2024: 36; 2025: 30; 2026: 10.

110 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  4. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  7. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  8. 2022Patching open-vocabulary models by interpolating weightscited by 187
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  10. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  11. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  12. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
38 more
  1. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  2. 2024Model merging with SVD to tie the Knotscited by 118
  3. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  4. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  6. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  7. 2023Deep Model Fusion: A Surveycited by 89
  8. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  9. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  10. 2024Parameter Competition Balancing for Model Mergingcited by 71
  11. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  12. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  13. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  14. 2024What Matters for Model Merging at Scale?cited by 52
  15. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  16. 2024SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teachercited by 45
  17. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  18. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  19. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  20. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  21. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  22. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  23. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  24. 2024Revisiting Weight Averaging for Model Mergingcited by 24
  25. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  26. 2024Here's a Free Lunch: Sanitizing Backdoored Models with Model Mergecited by 21
  27. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  28. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  29. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  30. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  31. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  32. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  33. 2025Update Your Transformer to the Latest Release: Re-Basin of Task Vectorscited by 17
  34. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  35. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  36. 2023Git-Theta: A Git Extension for Collaborative Development of Machine Learning Modelscited by 15
  37. 2024FedFisher: Leveraging Fisher Information for One-Shot Federated Learningcited by 15
  38. 2023Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?cited by 13

50 most cited of 110 All 110 (JSON)