mlsanityData to 7 Oct 2026

Improving Generalization with Flat Hilbert Bayesian Inference

arXiv 2410.04196 ↗First posted 5 Oct 2024cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce Flat Hilbert Bayesian Inference (FHBI), an algorithm designed to enhance generalization in Bayesian inference. Our approach involves an iterative two-step procedure with an adversarial functional perturbation step and a functional descent step within a reproducing kernel Hilbert space. This methodology is supported by a theoretical analysis that extends previous findings on generalization ability from finite-dimensional Euclidean spaces to infinite-dimensional functional spaces. To evaluate the effectiveness of FHBI, we conduct comprehensive comparisons against nine baseline methods on the textttVTAB-1K benchmark, which encompasses 19 diverse datasets across various domains with diverse semantics. Empirical results demonstrate that FHBI consistently outperforms the baselines by notable margins, highlighting its practical efficacy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

37 of 54 references.

References · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Exploring Generalization in Deep Learningcited by 595
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  12. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
25 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  2. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  3. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  4. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  7. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  8. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  9. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  10. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  11. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  12. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  13. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  14. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  15. 2020Relative Flatness and Generalizationcited by 88
  16. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  17. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  18. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  19. 2022SAM as an Optimal Relaxation of Bayescited by 40
  20. 2018Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priorscited by 35
  21. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  22. 2022Towards Bridging the Performance Gaps of Joint Energy-based Modelscited by 15
  23. 2023Flat Seeking Bayesian Neural Networkscited by 10
  24. 2019Walsh-Hadamard Variational Inference for Bayesian Deep Learningcited by 8
  25. 2024Neural Operators for Nonlinear Functionals on RKHScited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  2. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  3. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  4. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  5. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1