# Simple and Effective VAE Training with Calibrated Decoders

arXiv: https://arxiv.org/abs/2006.13202 · HTML: https://mlsanity.com/paper/2006.13202/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 30 of 57)

- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Space](https://mlsanity.com/paper/2503.15451/) (2025)
- [PoseFix: Correcting 3D Human Poses with Natural Language](https://mlsanity.com/paper/2309.08480/) (2023)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [PoseScript: Linking 3D Human Poses and Natural Language](https://mlsanity.com/paper/2210.11795/) (2022)
- [Long-term Causal Effects Estimation via Latent Surrogates Representation Learning](https://mlsanity.com/paper/2208.04589/) (2022)
- [Topic Modeling as Multi-Objective Contrastive Optimization](https://mlsanity.com/paper/2402.07577/) (2024)
- [Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11327/) (2021)
- [Beta-VAE Reproducibility: Challenges and Extensions](https://mlsanity.com/paper/2112.14278/) (2021)
- [Generating Multivariate Load States Using a Conditional Variational Autoencoder](https://mlsanity.com/paper/2110.11435/) (2021)
- [Stochastic Video Prediction with Structure and Motion](https://mlsanity.com/paper/2203.10528/) (2022)
- [Camera-Conditioned Stable Feature Generation for Isolated Camera Supervised Person Re-IDentification](https://mlsanity.com/paper/2203.15210/) (2022)
- [Planning with RL and episodic-memory behavioral priors](https://mlsanity.com/paper/2207.01845/) (2022)
- [Micro and Macro Level Graph Modeling for Graph Variational Auto-Encoders](https://mlsanity.com/paper/2210.16844/) (2022)
- [Gromov-Wasserstein Autoencoders](https://mlsanity.com/paper/2209.07007/) (2022)
- [Re-parameterizing VAEs for stability](https://mlsanity.com/paper/2106.13739/) (2021)
- [PriorCVAE: scalable MCMC parameter inference with Bayesian deep generative modelling](https://mlsanity.com/paper/2304.04307/) (2023)
- [Matching aggregate posteriors in the variational autoencoder](https://mlsanity.com/paper/2311.07693/) (2023)
- [High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve](https://mlsanity.com/paper/2309.07663/) (2023)
- [Distinguishing representational geometries with controversial stimuli: Bayesian experimental design and its application to face dissimilarity judgments](https://mlsanity.com/paper/2211.15053/) (2022)
- [Remedying uncertainty representations in visual inference through Explaining-Away Variational Autoencoders](https://mlsanity.com/paper/2404.15390/) (2024)
- [Towards diffusion models for large-scale sea-ice modelling](https://mlsanity.com/paper/2406.18417/) (2024)
- [Lung-DDPM+: Efficient Thoracic CT Image Synthesis using Diffusion Probabilistic Model](https://mlsanity.com/paper/2508.09327/) (2025)
- [Deep Generative Domain Adaptation with Temporal Relation Knowledge for Cross-User Activity Recognition](https://mlsanity.com/paper/2403.14682/) (2024)
- [Deep Generative Domain Adaptation with Temporal Attention for Cross-User Activity Recognition](https://mlsanity.com/paper/2403.17958/) (2024)
- [Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm](https://mlsanity.com/paper/2408.05834/) (2024)
- [Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis](https://mlsanity.com/paper/2506.00849/) (2025)
- [VIVAT: Virtuous Improving VAE Training through Artifact Mitigation](https://mlsanity.com/paper/2506.07863/) (2025)

## Abstract (arXiv)

Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of image and video datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method. Project website is at https://orybkin.github.io/sigma-vae/