mlsanityData to 7 Oct 2026

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

arXiv 2203.05482 ↗First posted 10 Mar 2022cs.LG cs.CL cs.CVcited by 1,366

Not on the map. Why?

Cited by 59 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.

Key facts. Cited by 1,366 papers in the map (data through 7 Oct 2026). Builds on 77 of its 119 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

77 of 119 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017On Calibration of Modern Neural Networkscited by 4,860
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2019Natural Adversarial Examplescited by 1,401
  13. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  14. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  15. 2014How transferable are features in deep neural networks?cited by 1,216
  16. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  17. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  19. 2021Scaling Vision Transformerscited by 993
  20. 2018Neural Network Acceptability Judgmentscited by 952
  21. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  22. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  23. 2021Robust fine-tuning of zero-shot modelscited by 785
  24. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  25. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  26. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  27. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  28. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  29. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  30. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  31. 2018Do Better ImageNet Models Transfer Better?cited by 560
  32. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  33. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  34. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  35. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  36. 2020What is being transferred in transfer learning?cited by 365
  37. 2017Functional Map of the Worldcited by 338
  38. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 87; 2023: 196; 2024: 332; 2025: 369; 2026: 382.

1,366 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022Editing Models with Task Arithmeticcited by 1,076
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  6. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  7. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  9. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
38 more
  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  3. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  4. 2024The Platonic Representation Hypothesiscited by 362
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  6. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 289
  7. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  8. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  9. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  11. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  13. 2023Nash Learning from Human Feedbackcited by 220
  14. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  15. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  16. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  17. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  18. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  19. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  20. 2022Patching open-vocabulary models by interpolating weightscited by 187
  21. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  22. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  23. 2023Mitigating the Alignment Tax of RLHFcited by 181
  24. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  25. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  26. 2024A Roadmap to Pluralistic Alignmentcited by 175
  27. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  28. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  29. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  30. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  31. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  32. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  33. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  34. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  35. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  36. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  37. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  38. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130

50 most cited of 1,366 All 1,366 (JSON)