# Beta-Sigma VAE: Separating beta and decoder variance in Gaussian variational autoencoder

arXiv: https://arxiv.org/abs/2409.09361 · HTML: https://mlsanity.com/paper/2409.09361/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 26 of 26)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [Structured Disentangled Representations](https://mlsanity.com/paper/1804.02086/) (2018)
- [Simple and Effective VAE Training with Calibrated Decoders](https://mlsanity.com/paper/2006.13202/) (2020)
- [Loss Landscapes of Regularized Linear Autoencoders](https://mlsanity.com/paper/1901.08168/) (2019)
- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [Leveraging the Exact Likelihood of Deep Latent Variable Models](https://mlsanity.com/paper/1802.04826/) (2018)
- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [Evaluating Lossy Compression Rates of Deep Generative Models](https://mlsanity.com/paper/2008.06653/) (2020)
- [VAE Approximation Error: ELBO and Exponential Families](https://mlsanity.com/paper/2102.09310/) (2021)
- [A Tutorial on VAEs: From Bayes' Rule to Lossless Compression](https://mlsanity.com/paper/2006.10273/) (2020)
- [Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias](https://mlsanity.com/paper/2112.06868/) (2021)

## Abstract (arXiv)

Variational autoencoder (VAE) is an established generative model but is notorious for its blurriness. In this work, we investigate the blurry output problem of VAE and resolve it, exploiting the variance of Gaussian decoder and β of beta-VAE. Specifically, we reveal that the indistinguishability of decoder variance and β hinders appropriate analysis of the model by random likelihood value, and limits performance improvement by omitting the gain from β. To address the problem, we propose Beta-Sigma VAE (BS-VAE) that explicitly separates β and decoder variance σ2x in the model. Our method demonstrates not only superior performance in natural image synthesis but also controllable parameters and predictable analysis compared to conventional VAE. In our experimental evaluation, we employ the analysis of rate-distortion curve and proxy metrics on computer vision datasets. The code is available on https://github.com/overnap/BS-VAE