mlsanityData to 14 Oct 2026

SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks

arXiv 2606.23357 ↗First posted 22 Jun 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addressed by adapting the SOAP optimizer. Our key idea is to run IVON, an existing diagonal-covariance variational method, in the eigenspace of SOAP's preconditioner and then use the preconditioner to transform the diagonal estimate into a non-diagonal covariance. The resulting method has costs similar to those of SOAP and requires no drastic changes to training pipelines. We call the posteriors obtained in this way SOAP-Bubbles and our new optimizer Eigenspace-VON (EVON). We show that, for logistic regression, EVON recovers the exact Gaussian covariance and that, for language model pretraining, it yields significantly better results than existing diagonal-covariance methods. Our work makes it easier to estimate more expressive posterior distributions for deep learning at scale.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

43 of 65 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  10. 2017Variational Continual Learningcited by 339
  11. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  12. 2019Practical Deep Learning with Bayesian Principlescited by 158
31 more
  1. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  2. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  3. 2021The Bayesian Learning Rulecited by 99
  4. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  5. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  6. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  7. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  8. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  9. 2024A New Perspective on Shampoo's Preconditionercited by 59
  10. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  11. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  12. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  13. 2018SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradientcited by 34
  14. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 32
  15. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  16. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 28
  17. 2019Validated Variational Inference via Practical Posterior Error Boundscited by 26
  18. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  19. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  20. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  21. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  22. 2024Bayesian Online Natural Gradient (BONG)cited by 14
  23. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 11
  24. 2018Eigenvalue Corrected Noisy Natural Gradientcited by 7
  25. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  26. 2025Knowledge Adaptation as Posterior Correctioncited by 6
  27. 2025Improving LoRA with Variational Learningcited by 5
  28. 2024Noether's razor: Learning Conserved Quantitiescited by 4
  29. 2025Federated ADMM from Bayesian Dualitycited by 3
  30. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 3
  31. 2025VIKING: Deep variational inference with stochastic projectionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Parameter Exploration for RLVR via Variational Learningcited by 1
  2. 2026Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogatecited by 0
  3. 2026Train for Many, Update with One: Improving Multi-Sample Reasoning via Fenchel-Bregman Posterior Learningcited by 0