mlsanityData to 7 Oct 2026

The continuous Bernoulli: fixing a pervasive error in variational autoencoders

arXiv 1907.06845 ↗First posted 16 Jul 2019stat.ML cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Variational autoencoders (VAE) have quickly become a central tool in machine learning, applicable to a broad range of data types and latent variable models. By far the most common first step, taken by seminal papers and by core software libraries alike, is to model MNIST data using a deep network parameterizing a Bernoulli likelihood. This practice contains what appears to be and what is often set aside as a minor inconvenience: the pixel data is [0,1] valued, not {0,1} as supported by the Bernoulli likelihood. Here we show that, far from being a triviality or nuisance that is convenient to ignore, this error has profound importance to VAE, both qualitative and quantitative. We introduce and fully characterize a new [0,1]-supported, single parameter distribution: the continuous Bernoulli, which patches this pervasive bug in VAE. This distribution is not nitpicking; it produces meaningful performance improvements across a range of metrics and datasets, including sharper image samples, and suggests a broader class of performant VAE.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 9; 2023: 9; 2024: 4; 2025: 4; 2026: 5.

40 papers.

Sort:

By area:

  1. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  2. 2021Neural Feature Search for RGB-Infrared Person Re-Identificationcited by 50
  3. 2022Diagnosing and Fixing Manifold Overfitting in Deep Generative Modelscited by 28
  4. 2025SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Predictioncited by 21
  5. 2022Instance-Dependent Noisy Label Learning via Graphical Modellingcited by 17
  6. 2021Certifiably Robust Variational Autoencoderscited by 14
  7. 2021Nested Variational Inferencecited by 14
  8. 2024Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcriptioncited by 14
  9. 2021Model Selection for Bayesian Autoencoderscited by 10
  10. 2021Robust outlier detection by de-biasing VAE likelihoodscited by 8
  11. 2023Extraction and Recovery of Spatio-Temporal Structure in Latent Dynamics Alignment with Diffusion Modelscited by 7
  12. 2022Energy-Based Models for Functional Data using Path Measure Tiltingcited by 5
28 more
  1. 2022FretNet: Continuous-Valued Pitch Contour Streaming for Polyphonic Guitar Tablature Transcriptioncited by 5
  2. 2023Variational Mixture of HyperGenerators for Learning Distributions Over Functionscited by 5
  3. 2021Probabilistic task modelling for meta-learningcited by 4
  4. 2021Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detectioncited by 4
  5. 2022Deep equilibrium models as estimators for continuous latent variablescited by 4
  6. 2023Real-Time Variational Method for Learning Neural Trajectory and its Dynamicscited by 3
  7. 2023Extreme heatwave sampling and prediction with analog Markov chain and comparisons with deep learningcited by 3
  8. 2024Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithmcited by 3
  9. 2024Stabilizing the Kumaraswamy Distributioncited by 3
  10. 2021Sparse Communication via Mixed Distributionscited by 2
  11. 2022Training Latent Variable Models with Auto-encoding Variational Bayes: A Tutorialcited by 2
  12. 2023A behavioural transformer for effective collaboration between a robot and a non-stationary humancited by 2
  13. 2022DBCal: Density Based Calibration of classifier predictions for uncertainty quantificationcited by 1
  14. 2022On the Normalizing Constant of the Continuous Categorical Distributioncited by 1
  15. 2023Anomaly Detection in Aeronautics Data with Quantum-compatible Discrete Deep Generative Modelcited by 1
  16. 2023Self-Distillation for Gaussian Process Regression and Classificationcited by 1
  17. 2025An Introduction to Discrete Variational Autoencoderscited by 1
  18. 2022Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Modelcited by 0
  19. 2023On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Modelscited by 0
  20. 2023Auto-encoding GPS data to reveal individual and collective behaviourcited by 0
  21. 2024Towards Model-Agnostic Posterior Approximation for Fast and Accurate Variational Autoencoderscited by 0
  22. 2025Generative Binary Memory: Pseudo-Replay Class-Incremental Learning on Binarized Embeddingscited by 0
  23. 2025Adaptive Heterogeneous Mixtures of Normalising Flows for Robust Variational Inferencecited by 0
  24. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  25. 2026Predicting integers from continuous parameterscited by 0
  26. 2026Metabolic cost of information processing in Poisson variational autoencoderscited by 0
  27. 2026Variational Learning of Fractional Posteriorscited by 0
  28. 2026Lost and Found in Translation: Variational Diagnostics for Neural Codebook Channelscited by 0