mlsanityData to 7 Oct 2026

Variational Learning is Effective for Large Deep Networks

arXiv 2402.17641 ↗First posted 27 Feb 2024cs.LG cs.AI cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve finetuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence that variational learning is effective.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 47 of its 77 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

47 of 77 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
35 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018Neural Network Acceptability Judgmentscited by 952
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2015Weight Uncertainty in Neural Networkscited by 554
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  13. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  14. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  15. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  16. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  17. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  18. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  19. 2019Practical Deep Learning with Bayesian Principlescited by 158
  20. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  21. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  22. 2021Bayesian Neural Network Priors Revisitedcited by 121
  23. 2021The Bayesian Learning Rulecited by 95
  24. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  25. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  26. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  27. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  28. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  29. 2015Equilibrated adaptive learning rates for non-convex optimizationcited by 42
  30. 2022SAM as an Optimal Relaxation of Bayescited by 40
  31. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  32. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  33. 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
  34. 2018Overpruning in Variational Bayesian Neural Networkscited by 21
  35. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 26; 2026: 20.

62 papers.

Sort:

By area:

  1. 2024(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Modelscited by 15
  2. 2024Bayesian Online Natural Gradient (BONG)cited by 14
  3. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  4. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 13
  5. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  6. 2025CLEAR: Calibrated Learning for Epistemic and Aleatoric Riskcited by 6
  7. 2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 6
  8. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  9. 2024Variational Low-Rank Adaptation Using IVONcited by 5
  10. 2024Streamlining Prediction in Bayesian Deep Learningcited by 5
  11. 2025Variational Visual Question Answering for Uncertainty-Aware Selective Predictioncited by 5
  12. 2025Improving LoRA with Variational Learningcited by 5
38 more
  1. 2024Inclusive KL Gradient Flows: Otto-Wasserstein, Fisher-Rao-Gaussian, and Local-Estimator Dynamicscited by 4
  2. 2025Connecting Federated ADMM to Bayescited by 4
  3. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
  4. 2025Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction taskscited by 3
  5. 2025Federated ADMM from Bayesian Dualitycited by 3
  6. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  7. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
  8. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  9. 2024Evaluating Bayesian deep learning for radio galaxy classificationcited by 2
  10. 2024A Bayesian Interpretation of Adaptive Low-Rank Adaptationcited by 2
  11. 2025Asymmetric Duos: Sidekicks Improve Uncertaintycited by 2
  12. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  13. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2
  14. 2025Information Geometry of Variational Bayescited by 2
  15. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  16. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  17. 2024Federated Learning with Uncertainty and Personalization via Efficient Second-order Optimizationcited by 1
  18. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  19. 2024Information-Geometric Barycenters for Bayesian Federated Learningcited by 1
  20. 2025Fenchel-Young Variational Learningcited by 1
  21. 2025How To Make Your Cell Tracker Say "I dunno!"cited by 1
  22. 2025Quantifying the uncertainty of model-based synthetic image quality metricscited by 1
  23. 2025Scaling Laws for Uncertainty in Deep Learningcited by 1
  24. 2025Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inferencecited by 1
  25. 2026Gradient Regularized Natural Gradientscited by 1
  26. 2026A Stein Identity for q-Gaussians with Bounded Supportcited by 1
  27. 2026Calibrating Scientific Foundation Models with Inference-Time Stochastic Attentioncited by 1
  28. 2026Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learningcited by 1
  29. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 1
  30. 2026Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphscited by 1
  31. 2024Variation Due to Regularization Tractably Recovers Bayesian Deep Learningcited by 0
  32. 2025Spectral-factorized Positive-definite Curvature Learning for NN Trainingcited by 0
  33. 2025Variational Learning Induces Adaptive Label Smoothingcited by 0
  34. 2025Variational Bayesian Pseudo-Coresetcited by 0
  35. 2025On Local Posterior Structure in Deep Ensemblescited by 0
  36. 2025Variational Deep Learning via Implicit Regularizationcited by 0
  37. 2025Cost-Free Personalization via Information-Geometric Projection in Bayesian Federated Learningcited by 0
  38. 2025SVRG and Beyond via Posterior Correctioncited by 0

50 most cited of 62 All 62 (JSON)