# On Implicit Regularization in β-VAEs

arXiv: https://arxiv.org/abs/2002.00041 · HTML: https://mlsanity.com/paper/2002.00041/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 30 of 34)

- [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [Function Classes for Identifiable Nonlinear Independent Component Analysis](https://mlsanity.com/paper/2208.06406/) (2022)
- [Score-based Causal Representation Learning with Interventions](https://mlsanity.com/paper/2301.08230/) (2023)
- [Can Push-forward Generative Models Fit Multimodal Distributions?](https://mlsanity.com/paper/2206.14476/) (2022)
- [General Identifiability and Achievability for Causal Representation Learning](https://mlsanity.com/paper/2310.15450/) (2023)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [A Geometric Perspective on Variational Autoencoders](https://mlsanity.com/paper/2209.07370/) (2022)
- [On PAC-Bayesian reconstruction guarantees for VAEs](https://mlsanity.com/paper/2202.11455/) (2022)
- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization](https://mlsanity.com/paper/2106.02923/) (2021)
- [RAPiD-Seg: Range-Aware Pointwise Distance Distribution Networks for 3D LiDAR Segmentation](https://mlsanity.com/paper/2407.10159/) (2024)
- [Certifiably Robust Variational Autoencoders](https://mlsanity.com/paper/2102.07559/) (2021)
- [Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias](https://mlsanity.com/paper/2112.06868/) (2021)
- [Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation](https://mlsanity.com/paper/2302.11294/) (2023)
- [Mapping the Multiverse of Latent Representations](https://mlsanity.com/paper/2402.01514/) (2024)
- [SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTA](https://mlsanity.com/paper/2303.16666/) (2023)
- [Implicit Regularization with Polynomial Growth in Deep Tensor Factorization](https://mlsanity.com/paper/2207.08942/) (2022)
- [Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors](https://mlsanity.com/paper/2210.01760/) (2022)
- [Ensuring Topological Data-Structure Preservation under Autoencoder Compression due to Latent Space Regularization in Gauss--Legendre nodes](https://mlsanity.com/paper/2309.08228/) (2023)
- [On Kernel-based Variational Autoencoder](https://mlsanity.com/paper/2405.12783/) (2024)
- [Content-Style Identification via Differential Independence](https://mlsanity.com/paper/2605.17827/) (2026)
- [Reproducible, incremental representation learning with Rosetta VAE](https://mlsanity.com/paper/2201.05206/) (2022)
- [Flat Latent Manifolds for Human-machine Co-creation of Music](https://mlsanity.com/paper/2202.12243/) (2022)
- [Interval Bound Interpolation for Few-shot Learning with Few Tasks](https://mlsanity.com/paper/2204.03511/) (2022)
- [TimewarpVAE: Simultaneous Time-Warping and Representation Learning of Trajectories](https://mlsanity.com/paper/2310.16027/) (2023)
- [Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training](https://mlsanity.com/paper/2402.01238/) (2024)
- [Flexible Bayesian Last Layer Models Using Implicit Priors and Diffusion Posterior Sampling](https://mlsanity.com/paper/2408.03746/) (2024)
- [Enhancing Interpretability in Generative Modeling: Statistically Disentangled Latent Spaces Guided by Generative Factors in Scientific Datasets](https://mlsanity.com/paper/2507.00298/) (2025)

## Abstract (arXiv)

While the impact of variational inference (VI) on posterior inference in a fixed generative model is well-characterized, its role in regularizing a learned generative model when used in variational autoencoders (VAEs) is poorly understood. We study the regularizing effects of variational distributions on learning in generative models from two perspectives. First, we analyze the role that the choice of variational family plays in imparting uniqueness to the learned model by restricting the set of optimal generative models. Second, we study the regularization effect of the variational family on the local geometry of the decoding model. This analysis uncovers the regularizer implicit in the β-VAE objective, and leads to an approximation consisting of a deterministic autoencoding objective plus analytic regularizers that depend on the Hessian or Jacobian of the decoding model, unifying VAEs with recent heuristics proposed for training regularized autoencoders. We empirically verify these findings, observing that the proposed deterministic objective exhibits similar behavior to the β-VAE in terms of objective value and sample quality.