# A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

arXiv: https://arxiv.org/abs/1602.08734 · HTML: https://mlsanity.com/paper/1602.08734/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2016.

## Built on by (most cited first, 5 of 5)

- [Variational Sparse Coding with Learned Thresholding](https://mlsanity.com/paper/2205.03665/) (2022)
- [CARNet: A Dynamic Autoencoder for Learning Latent Dynamics in Autonomous Driving Tasks](https://mlsanity.com/paper/2205.08712/) (2022)
- [KARNet: Kalman Filter Augmented Recurrent Neural Network for Learning World Models in Autonomous Driving Tasks](https://mlsanity.com/paper/2305.14644/) (2023)
- [Autocodificadores Variacionales (VAE) Fundamentos Te'oricos y Aplicaciones](https://mlsanity.com/paper/2302.09363/) (2023)
- [Your Image is Secretly the Last Frame of a Pseudo Video](https://mlsanity.com/paper/2410.20158/) (2024)

## Abstract (arXiv)

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability necessary for efficient stochastic gradient variational inference. To learn the parameters of the new model, we approximate the posterior of the latent variables with a variational auto-encoder. Rather than making the usual mean-field assumption however, the encoder parameterizes a new type of structured variational approximation that retains the prior dependencies of the generative model. Using this structured posterior approximation, we are able to perform joint training of deep models with many layers of latent random variables, without having to resort to stacking or other layerwise training procedures.