# Analytically Tractable Inference in Deep Neural Networks

arXiv: https://arxiv.org/abs/2103.05461 · HTML: https://mlsanity.com/paper/2103.05461/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Convolutional Neural Networks at Constrained Time Cost](https://mlsanity.com/paper/1412.1710/) (2014)
- [Probabilistic Binary Neural Networks](https://mlsanity.com/paper/1809.03368/) (2018)
- [Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes](https://mlsanity.com/paper/1611.04520/) (2016)
- [Tractable Approximate Gaussian Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2004.09281/) (2020)

## Built on by (most cited first, 3 of 3)

- [Analytically Tractable Bayesian Deep Q-Learning](https://mlsanity.com/paper/2106.11086/) (2021)
- [Analytically Tractable Hidden-States Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2107.03759/) (2021)
- [Distillation of a tractable model from the VQ-VAE](https://mlsanity.com/paper/2509.01400/) (2025)

## Abstract (arXiv)

Since its inception, deep learning has been overwhelmingly reliant on backpropagation and gradient-based optimization algorithms in order to learn weight and bias parameter values. Tractable Approximate Gaussian Inference (TAGI) algorithm was shown to be a viable and scalable alternative to backpropagation for shallow fully-connected neural networks. In this paper, we are demonstrating how TAGI matches or exceeds the performance of backpropagation, for training classic deep neural network architectures. Although TAGI's computational efficiency is still below that of deterministic approaches relying on backpropagation, it outperforms them on classification tasks and matches their performance for information maximizing generative adversarial networks while using smaller architectures trained with fewer epochs.