# Learning Discrete Distributions by Dequantization

arXiv: https://arxiv.org/abs/2001.11235 · HTML: https://mlsanity.com/paper/2001.11235/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 18 of 18)

- [iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression](https://mlsanity.com/paper/2103.16211/) (2021)
- [iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder](https://mlsanity.com/paper/2111.00965/) (2021)
- [Invertible DenseNets with Concatenated LipSwish](https://mlsanity.com/paper/2102.02694/) (2021)
- [Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation](https://mlsanity.com/paper/2109.14885/) (2021)
- [Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Prior](https://mlsanity.com/paper/2203.00479/) (2022)
- [CeFlow: A Robust and Efficient Counterfactual Explanation Framework for Tabular Data using Normalizing Flows](https://mlsanity.com/paper/2303.14668/) (2023)
- [Manifold Density Estimation via Generalized Dequantization](https://mlsanity.com/paper/2102.07143/) (2021)
- [Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives](https://mlsanity.com/paper/2201.10989/) (2022)
- [AdaCat: Adaptive Categorical Discretization for Autoregressive Models](https://mlsanity.com/paper/2208.02246/) (2022)
- [Simplex-to-Euclidean Bijections for Categorical Flow Matching](https://mlsanity.com/paper/2510.27480/) (2025)
- [Flow Matching for Count Data](https://mlsanity.com/paper/2605.07746/) (2026)
- [Learning to Bound: A Generative Cram'er-Rao Bound](https://mlsanity.com/paper/2203.03695/) (2022)
- [Mixed Variational Flows for Discrete Variables](https://mlsanity.com/paper/2308.15613/) (2023)
- [Model-Based Counterfactual Explanations Incorporating Feature Space Attributes for Tabular Data](https://mlsanity.com/paper/2404.13224/) (2024)
- [OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression](https://mlsanity.com/paper/2111.01662/) (2021)
- [BabyFlow: 3D modeling of realistic and expressive infant faces](https://mlsanity.com/paper/2512.19560/) (2025)
- [A Generative Sampler for distributions with possible discrete parameter based on Reversibility](https://mlsanity.com/paper/2603.09251/) (2026)
- [Belief Embedding Tree Search](https://mlsanity.com/paper/2610.15046/) (2026)

## Abstract (arXiv)

Media is generally stored digitally and is therefore discrete. Many successful deep distribution models in deep learning learn a density, i.e., the distribution of a continuous random variable. Na"ive optimization on discrete data leads to arbitrarily high likelihoods, and instead, it has become standard practice to add noise to datapoints. In this paper, we present a general framework for dequantization that captures existing methods as a special case. We derive two new dequantization objectives: importance-weighted (iw) dequantization and R'enyi dequantization. In addition, we introduce autoregressive dequantization (ARD) for more flexible dequantization distributions. Empirically we find that iw and R'enyi dequantization considerably improve performance for uniform dequantization distributions. ARD achieves a negative log-likelihood of 3.06 bits per dimension on CIFAR10, which to the best of our knowledge is state-of-the-art among distribution models that do not require autoregressive inverses for sampling.