# Deep Data Density Estimation through Donsker-Varadhan Representation

arXiv: https://arxiv.org/abs/2104.06612 · HTML: https://mlsanity.com/paper/2104.06612/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [High-Dimensional Probability Estimation with Deep Density Models](https://mlsanity.com/paper/1302.5125/) (2013)
- [Interpreting Rate-Distortion of Variational Autoencoder and Using Model Uncertainty for Anomaly Detection](https://mlsanity.com/paper/2005.01889/) (2020)
- [Combining Stochastic Adaptive Cubic Regularization with Negative Curvature for Nonconvex Optimization](https://mlsanity.com/paper/1906.11417/) (2019)

## Built on by (most cited first, 6 of 6)

- [Confidence-Aware Graph Neural Networks for Learning Reliability Assessment Commitments](https://mlsanity.com/paper/2211.15755/) (2022)
- [Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning](https://mlsanity.com/paper/2507.14748/) (2025)
- [REMEDI: Corrective Transformations for Improved Neural Entropy Estimation](https://mlsanity.com/paper/2402.05718/) (2024)
- [TREET: TRansfer Entropy Estimation via Transformers](https://mlsanity.com/paper/2402.06919/) (2024)
- [Conformalised data synthesis](https://mlsanity.com/paper/2312.08999/) (2023)
- [NMINE: Normalized Mutual Information Neural Estimation](https://mlsanity.com/paper/2607.27710/) (2026)

## Abstract (arXiv)

Estimating the data density is one of the challenging problems in deep learning. In this paper, we present a simple yet effective method for estimating the data density using a deep neural network and the Donsker-Varadhan variational lower bound on the KL divergence. We show that the optimal critic function associated with the Donsker-Varadhan representation on the KL divergence between the data and the uniform distribution can estimate the data density. We also present the deep neural network-based modeling and its stochastic learning. The experimental results and possible applications of the proposed method demonstrate that it is competitive with the previous methods and has a lot of possibilities in applied to various applications.