# From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network

arXiv: https://arxiv.org/abs/2512.13111 · HTML: https://mlsanity.com/paper/2512.13111/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Subspace Inference for Bayesian Deep Learning](https://mlsanity.com/paper/1907.07504/) (2019)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Riemannian Laplace approximations for Bayesian neural networks](https://mlsanity.com/paper/2306.07158/) (2023)
- [The Peril of Popular Deep Learning Uncertainty Estimation Methods](https://mlsanity.com/paper/2112.05000/) (2021)
- [Kalman Bayesian Neural Networks for Closed-form Online Learning](https://mlsanity.com/paper/2110.00944/) (2021)
- [Tractable Approximate Gaussian Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2004.09281/) (2020)
- [Batch Reinforcement Learning on the Industrial Benchmark: First Experiences](https://mlsanity.com/paper/1705.07262/) (2017)
- [Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priors](https://mlsanity.com/paper/2207.08200/) (2022)
- [Student-t processes as infinite-width limits of posterior Bayesian neural networks](https://mlsanity.com/paper/2502.04247/) (2025)

## Abstract (arXiv)

In recent years, neural networks have revolutionized various domains, yet challenges such as hyperparameter tuning and overfitting remain significant hurdles. Bayesian neural networks offer a framework to address these challenges by incorporating uncertainty directly into the model, yielding more reliable predictions, particularly for out-of-distribution data. This paper presents Hierarchical Approximate Bayesian Neural Network, a novel approach that uses a Gaussian-inverse-Wishart distribution as a hyperprior of the network's weights to increase both the robustness and performance of the model. We provide analytical representations for the predictive distribution and weight posterior, which amount to the calculation of the parameters of Student's t-distributions in closed form with linear complexity with respect to the number of weights. Our method demonstrates robust performance, effectively addressing issues of overfitting and providing reliable uncertainty estimates, particularly for out-of-distribution tasks. Experimental results indicate that HABNN not only matches but often outperforms state-of-the-art models, suggesting a promising direction for future applications in safety-critical environments.