# Variance Networks: When Expectation Does Not Meet Your Expectations

arXiv: https://arxiv.org/abs/1803.03764 · HTML: https://mlsanity.com/paper/1803.03764/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks](https://mlsanity.com/paper/2211.12717/) (2022)
- [Deep-Attack over the Deep Reinforcement Learning](https://mlsanity.com/paper/2205.00807/) (2022)
- [Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty](https://mlsanity.com/paper/2305.00934/) (2023)
- [Solving robust MDPs as a sequence of static RL problems](https://mlsanity.com/paper/2410.06212/) (2024)
- [Propagating Variational Model Uncertainty for Bioacoustic Call Label Smoothing](https://mlsanity.com/paper/2210.10526/) (2022)
- [Federated Learning via Meta-Variational Dropout](https://mlsanity.com/paper/2510.20225/) (2025)

## Abstract (arXiv)

Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than conventional parameterizations where the mean is being learned.