# The continuous Bernoulli: fixing a pervasive error in variational autoencoders

arXiv: https://arxiv.org/abs/1907.06845 · HTML: https://mlsanity.com/paper/1907.06845/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2019.

## Built on by (most cited first, 30 of 41)

- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Neural Feature Search for RGB-Infrared Person Re-Identification](https://mlsanity.com/paper/2104.02366/) (2021)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [Certifiably Robust Variational Autoencoders](https://mlsanity.com/paper/2102.07559/) (2021)
- [Nested Variational Inference](https://mlsanity.com/paper/2106.11302/) (2021)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [Model Selection for Bayesian Autoencoders](https://mlsanity.com/paper/2106.06245/) (2021)
- [Robust outlier detection by de-biasing VAE likelihoods](https://mlsanity.com/paper/2108.08760/) (2021)
- [Extraction and Recovery of Spatio-Temporal Structure in Latent Dynamics Alignment with Diffusion Models](https://mlsanity.com/paper/2306.06138/) (2023)
- [Energy-Based Models for Functional Data using Path Measure Tilting](https://mlsanity.com/paper/2202.01929/) (2022)
- [FretNet: Continuous-Valued Pitch Contour Streaming for Polyphonic Guitar Tablature Transcription](https://mlsanity.com/paper/2212.03023/) (2022)
- [Variational Mixture of HyperGenerators for Learning Distributions Over Functions](https://mlsanity.com/paper/2302.06223/) (2023)
- [Probabilistic task modelling for meta-learning](https://mlsanity.com/paper/2106.04802/) (2021)
- [Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection](https://mlsanity.com/paper/2107.13304/) (2021)
- [Deep equilibrium models as estimators for continuous latent variables](https://mlsanity.com/paper/2211.05943/) (2022)
- [Real-Time Variational Method for Learning Neural Trajectory and its Dynamics](https://mlsanity.com/paper/2305.11278/) (2023)
- [Extreme heatwave sampling and prediction with analog Markov chain and comparisons with deep learning](https://mlsanity.com/paper/2307.09060/) (2023)
- [Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm](https://mlsanity.com/paper/2408.05834/) (2024)
- [Stabilizing the Kumaraswamy Distribution](https://mlsanity.com/paper/2410.00660/) (2024)
- [Sparse Communication via Mixed Distributions](https://mlsanity.com/paper/2108.02658/) (2021)
- [Training Latent Variable Models with Auto-encoding Variational Bayes: A Tutorial](https://mlsanity.com/paper/2208.07818/) (2022)
- [A behavioural transformer for effective collaboration between a robot and a non-stationary human](https://mlsanity.com/paper/2307.13447/) (2023)
- [DBCal: Density Based Calibration of classifier predictions for uncertainty quantification](https://mlsanity.com/paper/2204.00150/) (2022)
- [On the Normalizing Constant of the Continuous Categorical Distribution](https://mlsanity.com/paper/2204.13290/) (2022)
- [Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model](https://mlsanity.com/paper/2208.03936/) (2022)
- [Anomaly Detection in Aeronautics Data with Quantum-compatible Discrete Deep Generative Model](https://mlsanity.com/paper/2303.12302/) (2023)
- [Self-Distillation for Gaussian Process Regression and Classification](https://mlsanity.com/paper/2304.02641/) (2023)
- [An Introduction to Discrete Variational Autoencoders](https://mlsanity.com/paper/2505.10344/) (2025)

## Abstract (arXiv)

Variational autoencoders (VAE) have quickly become a central tool in machine learning, applicable to a broad range of data types and latent variable models. By far the most common first step, taken by seminal papers and by core software libraries alike, is to model MNIST data using a deep network parameterizing a Bernoulli likelihood. This practice contains what appears to be and what is often set aside as a minor inconvenience: the pixel data is [0,1] valued, not {0,1} as supported by the Bernoulli likelihood. Here we show that, far from being a triviality or nuisance that is convenient to ignore, this error has profound importance to VAE, both qualitative and quantitative. We introduce and fully characterize a new [0,1]-supported, single parameter distribution: the continuous Bernoulli, which patches this pervasive bug in VAE. This distribution is not nitpicking; it produces meaningful performance improvements across a range of metrics and datasets, including sharper image samples, and suggests a broader class of performant VAE.