# Estimators of Entropy and Information via Inference in Probabilistic Models

arXiv: https://arxiv.org/abs/2202.12363 · HTML: https://mlsanity.com/paper/2202.12363/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [On the Quantitative Analysis of Decoder-Based Generative Models](https://mlsanity.com/paper/1611.04273/) (2016)
- [Inference Compilation and Universal Probabilistic Programming](https://mlsanity.com/paper/1610.09900/) (2016)
- [Deep Amortized Inference for Probabilistic Programs](https://mlsanity.com/paper/1610.05735/) (2016)
- [Sandwiching the marginal likelihood using bidirectional Monte Carlo](https://mlsanity.com/paper/1511.02543/) (2015)
- [Inference Networks for Sequential Monte Carlo in Graphical Models](https://mlsanity.com/paper/1602.06701/) (2016)
- [SPPL: Probabilistic Programming with Fast Exact Symbolic Inference](https://mlsanity.com/paper/2010.03485/) (2020)
- [GILBO: One Metric to Measure Them All](https://mlsanity.com/paper/1802.04874/) (2018)
- [Measuring the reliability of MCMC inference with bidirectional Monte Carlo](https://mlsanity.com/paper/1606.02275/) (2016)
- [AIDE: An algorithm for measuring the accuracy of probabilistic inference algorithms](https://mlsanity.com/paper/1705.07224/) (2017)
- [Detecting Dependencies in Sparse, Multivariate Databases Using Probabilistic Programming and Non-parametric Bayes](https://mlsanity.com/paper/1611.01708/) (2016)
- [Probabilistic Search for Structured Data via Probabilistic Programming and Nonparametric Bayes](https://mlsanity.com/paper/1704.01087/) (2017)

## Built on by (most cited first, 3 of 3)

- [Recursive Monte Carlo and Variational Inference with Auxiliary Variables](https://mlsanity.com/paper/2203.02836/) (2022)
- [Sequential Monte Carlo Learning for Time Series Structure Discovery](https://mlsanity.com/paper/2307.09607/) (2023)
- [Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacing](https://mlsanity.com/paper/2511.13602/) (2025)

## Abstract (arXiv)

Estimating information-theoretic quantities such as entropy and mutual information is central to many problems in statistics and machine learning, but challenging in high dimensions. This paper presents estimators of entropy via inference (EEVI), which deliver upper and lower bounds on many information quantities for arbitrary variables in a probabilistic generative model. These estimators use importance sampling with proposal distribution families that include amortized variational inference and sequential Monte Carlo, which can be tailored to the target model and used to squeeze true information values with high accuracy. We present several theoretical properties of EEVI and demonstrate scalability and efficacy on two problems from the medical domain: (i) in an expert system for diagnosing liver disorders, we rank medical tests according to how informative they are about latent diseases, given a pattern of observed symptoms and patient attributes; and (ii) in a differential equation model of carbohydrate metabolism, we find optimal times to take blood glucose measurements that maximize information about a diabetic patient's insulin sensitivity, given their meal and medication schedule.