Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Not on the map. Why?
Abstract (arXiv)
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 3 Feb 2025.
Builds on
40 of 48 references.
References · 40
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2023Mistral 7Bcited by 3,065
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2024Mixtral of Expertscited by 1,785
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2022Editing Models with Task Arithmeticcited by 1,076
28 more
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
- 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
- 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
- 2023Efficient Large Language Models: A Surveycited by 183
- 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
- 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
- 2022A Review of Sparse Expert Models in Deep Learningcited by 147
- 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
- 2022Fusing finetuned models for better pretrainingcited by 110
- 2024Language models scale reliably with over-training and on downstream taskscited by 105
- 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
- 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
- 2024Evaluating Quantized Large Language Modelscited by 75
- 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
- 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
- 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
- 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
- 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
- 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
- 2023Adaptive Training Distributions with Scalable Online Bilevel Optimizationcited by 11
- 2024Dynamic Gradient Alignment for Online Data Mixingcited by 10
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.