mlsanityData to 7 Oct 2026

Self-Soupervision: Cooking Model Soups without Labels

arXiv 2602.02890 ↗First posted 2 Feb 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (ImageNet-C) and +7% (LAION-C). Self-Soupervision also unlocks countless SSL algorithms to cook the diverse ingredients needed for more robust soups. We show for the first time that ingredients can differ in their SSL hyperparameters -- and more surprisingly, in their SSL algorithms. We cook soups of MAE, MoCoV3, MMCR, and LeJEPA ingredients that are more accurate than any single SSL ingredient.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

47 of 64 references.

References · 47

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2019Natural Adversarial Examplescited by 1,401
35 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  9. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  10. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  11. 2022DeiT III: Revenge of the ViTcited by 417
  12. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  13. 2020Are we done with ImageNet?cited by 329
  14. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  15. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  16. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  17. 2017Multi-task Self-Supervised Visual Learningcited by 200
  18. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  19. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  20. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  21. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  22. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  23. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  24. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  25. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 55
  26. 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
  27. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  28. 2023Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shiftscited by 23
  29. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
  30. 2022Under the Cover Infant Pose Estimation using Multimodal Datacited by 6
  31. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5
  32. 2025Soup-of-Experts: Pretraining Specialist Models via Parameters Averagingcited by 3
  33. 2025LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Modelscited by 2
  34. 2025DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretrainingcited by 2
  35. 2025LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soupscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learningcited by 0