# Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2209.11366 · HTML: https://mlsanity.com/paper/2209.11366/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [An overview on deep learning-based approximation methods for partial differential equations](https://mlsanity.com/paper/2012.12348/) (2020)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [f-Divergence Variational Inference](https://mlsanity.com/paper/2009.13093/) (2020)
- [Constraining Variational Inference with Geometric Jensen-Shannon Divergence](https://mlsanity.com/paper/2006.10599/) (2020)
- [Explanation and Use of Uncertainty Quantified by Bayesian Neural Network Classifiers for Breast Histopathology Images](https://mlsanity.com/paper/2010.12575/) (2020)
- [On the Robustness of Bayesian Neural Networks to Adversarial Attacks](https://mlsanity.com/paper/2207.06154/) (2022)
- [Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks](https://mlsanity.com/paper/2207.08435/) (2022)

## Built on by (most cited first, 6 of 6)

- [Bayesian neural networks for predicting uncertainty in full-field material response](https://mlsanity.com/paper/2406.14838/) (2024)
- [Text-to-3D Generation using Jensen-Shannon Score Distillation](https://mlsanity.com/paper/2503.10660/) (2025)
- [Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators](https://mlsanity.com/paper/2507.14652/) (2025)
- [A Bayesian Active Learning Approach to Comparative Judgement](https://mlsanity.com/paper/2308.13292/) (2023)
- [Two tales for a geometric Jensen--Shannon divergence](https://mlsanity.com/paper/2508.05066/) (2025)
- [Bayesian 3D Steerable CNNs: Enabling Equivariance and Uncertainty Quantification Simultaneously](https://mlsanity.com/paper/2606.15479/) (2026)

## Abstract (arXiv)

Bayesian neural networks (BNNs) are state-of-the-art machine learning methods that can naturally regularize and systematically quantify uncertainties using their stochastic parameters. Kullback-Leibler (KL) divergence-based variational inference used in BNNs suffers from unstable optimization and challenges in approximating light-tailed posteriors due to the unbounded nature of the KL divergence. To resolve these issues, we formulate a novel loss function for BNNs based on a new modification to the generalized Jensen-Shannon (JS) divergence, which is bounded. In addition, we propose a Geometric JS divergence-based loss, which is computationally efficient since it can be evaluated analytically. We found that the JS divergence-based variational inference is intractable, and hence employed a constrained optimization framework to formulate these losses. Our theoretical analysis and empirical experiments on multiple regression and classification data sets suggest that the proposed losses perform better than the KL divergence-based loss, especially when the data sets are noisy or biased. Specifically, there are approximately 5% and 8% improvements in accuracy for a noise-added CIFAR-10 dataset and a regression dataset, respectively. There is about a 13% reduction in false negative predictions of a biased histopathology dataset. In addition, we quantify and compare the uncertainty metrics for the regression and classification tasks.