mlsanityData to 7 Oct 2026

RADIN: Souping on a Budget

arXiv 2401.17790 ↗First posted 31 Jan 2024cs.LG cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Model Soups, extending Stochastic Weights Averaging (SWA), combine models fine-tuned with different hyperparameters. Yet, their adoption is hindered by computational challenges due to subset selection issues. In this paper, we propose to speed up model soups by approximating soups performance using averaged ensemble logits performances. Theoretical insights validate the congruence between ensemble logits and weight averaging soups across any mixing ratios. Our Resource ADjusted soups craftINg (RADIN) procedure stands out by allowing flexible evaluation budgets, enabling users to adjust his budget of exploration adapted to his resources while increasing performance at lower budget compared to previous greedy approach (up to 4% on ImageNet).

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 12 of its 22 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

12 of 22 references.

References · 12

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  4. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  5. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  6. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  7. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  8. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  9. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  10. 2023Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shiftscited by 23
  11. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  12. 2022Integral Continual Learning Along the Tangent Vector Field of Taskscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Enhanced Soups for Graph Neural Networkscited by 0