mlsanityData to 7 Oct 2026

ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning

arXiv 2212.01378 ↗First posted 2 Dec 2022cs.LG cs.CL cs.DCcited by 52

Not on the map. Why?

Abstract (arXiv)

We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion. It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the need for shared data. Consequentially, ColD Fusion can give rise to a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based upon. We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was trained on; and (b) is a better starting point for finetuning on unseen datasets. We show that ColD Fusion outperforms RoBERTa and even previous multitask models. Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points on average without any changes to the architecture.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 51 of its 91 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

51 of 91 references.

References · 51

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
38 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  11. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  12. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  13. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  14. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  15. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  16. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  17. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  18. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  19. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  20. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  21. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  22. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  23. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  24. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  25. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  26. 2022Fusing finetuned models for better pretrainingcited by 110
  27. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  28. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  29. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  30. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  31. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  32. 2022Mechanistic Mode Connectivitycited by 50
  33. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  34. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  35. 2017Emotion Intensities in Tweetscited by 32
  36. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  37. 2022Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experimentscited by 23
  38. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 20; 2024: 23; 2025: 5; 2026: 2.

52 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  9. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  10. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  11. 2024Model merging with SVD to tie the Knotscited by 118
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
38 more
  1. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  5. 2023Modular Deep Learningcited by 82
  6. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  7. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  8. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  9. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  10. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  11. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  12. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  13. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  14. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  15. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  16. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  17. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
  18. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  19. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  20. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  21. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  22. 2024Ensembling Diffusion Models via Adaptive Feature Aggregationcited by 12
  23. 2024ZipNN: Lossless Compression for AI Modelscited by 12
  24. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  25. 2024Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metricscited by 11
  26. 2024Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchorscited by 7
  27. 2025LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Mergingcited by 7
  28. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  29. 2024Lossless and Near-Lossless Compression for Foundation Modelscited by 5
  30. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  31. 2025CAMEx: Curvature-aware Merging of Expertscited by 4
  32. 2025CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamicscited by 4
  33. 2024DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Modelscited by 3
  34. 2024Foundation Model Engineering: Engineering Foundation Models Just as Engineering Softwarecited by 1
  35. 2024Improving the Classification Effect of Clinical Images of Diseases for Multi-Source Privacy Protectioncited by 1
  36. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  37. 2023Derivative Free Weight-space Ensemblingcited by 0
  38. 2023Cordyceps@LT-EDI: Patching Language-Specific Homophobia/Transphobia Classifiers with a Multilingual Understandingcited by 0

50 most cited of 52 All 52 (JSON)