# On Incorporating Inductive Biases into VAEs

arXiv: https://arxiv.org/abs/2106.13746 · HTML: https://mlsanity.com/paper/2106.13746/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 75 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)
- [Variational Sequential Monte Carlo](https://mlsanity.com/paper/1705.11140/) (2017)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Mixed-curvature Variational Autoencoders](https://mlsanity.com/paper/1911.08411/) (2019)

## Built on by (most cited first, 11 of 11)

- [Perspectives on Incorporating Expert Feedback into Model Updates](https://mlsanity.com/paper/2205.06905/) (2022)
- [Topological Obstructions and How to Avoid Them](https://mlsanity.com/paper/2312.07529/) (2023)
- [A Sparsity-promoting Dictionary Model for Variational Autoencoders](https://mlsanity.com/paper/2203.15758/) (2022)
- [Top-down inference in an early visual cortex inspired hierarchical Variational Autoencoder](https://mlsanity.com/paper/2206.00436/) (2022)
- [Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors](https://mlsanity.com/paper/2401.02739/) (2024)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)
- [Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck](https://mlsanity.com/paper/2203.02592/) (2022)
- [TC-VAE: Uncovering Out-of-Distribution Data Generative Factors](https://mlsanity.com/paper/2304.04103/) (2023)
- [Constructing VAE Latent Spaces with Prescribed Topology](https://mlsanity.com/paper/2606.07058/) (2026)
- [Multiple Invertible and Partial-Equivariant Function for Latent Vector Transformation to Enhance Disentanglement in VAEs](https://mlsanity.com/paper/2502.03740/) (2025)
- [Neural Operator-enabled Topology-informed Evolutionary Strategy for PDE-Constrained Optimization](https://mlsanity.com/paper/2607.07682/) (2026)

## Abstract (arXiv)

We explain why directly changing the prior can be a surprisingly ineffective mechanism for incorporating inductive biases into VAEs, and introduce a simple and effective alternative approach: Intermediary Latent Space VAEs(InteL-VAEs). InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these in turn to the latent representation using a parametric function that encapsulates our desired inductive bias(es). This allows us to impose properties like sparsity or clustering on learned representations, and incorporate human knowledge into the generative model. Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, InteL-VAEs are able to directly enforce desired characteristics. Moreover, they bypass the computation and encoder design issues caused by non-Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function. We show that these advantages, in turn, lead to both better generative models and better representations being learned.