# Neural Capacity Estimators: How Reliable Are They?

arXiv: https://arxiv.org/abs/2111.07401 · HTML: https://mlsanity.com/paper/2111.07401/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Deep Learning for Channel Coding via Neural Mutual Information Estimation](https://mlsanity.com/paper/1903.02865/) (2019)
- [Neural Entropic Estimation: A faster path to mutual information estimation](https://mlsanity.com/paper/1905.12957/) (2019)
- [Capacity of Continuous Channels with Memory via Directed Information Neural Estimator](https://mlsanity.com/paper/2003.04179/) (2020)
- [Joint Learning of Probabilistic and Geometric Shaping for Coded Modulation Systems](https://mlsanity.com/paper/2004.05062/) (2020)
- [Computing the Feedback Capacity of Finite State Channels using Reinforcement Learning](https://mlsanity.com/paper/2001.09685/) (2020)

## Built on by (most cited first, 4 of 4)

- [Data-Driven Optimization of Directed Information over Discrete Alphabets](https://mlsanity.com/paper/2301.00621/) (2023)
- [A Perspective on Neural Capacity Estimation: Viability and Reliability](https://mlsanity.com/paper/2203.11793/) (2022)
- [Data-Driven Estimation of Capacity Upper Bounds](https://mlsanity.com/paper/2205.06471/) (2022)
- [A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds](https://mlsanity.com/paper/2510.05386/) (2025)

## Abstract (arXiv)

Recently, several methods have been proposed for estimating the mutual information from sample data using deep neural networks and without the knowing closed form distribution of the data. This class of estimators is referred to as neural mutual information estimators. Although very promising, such techniques have yet to be rigorously bench-marked so as to establish their efficacy, ease of implementation, and stability for capacity estimation which is joint maximization frame-work. In this paper, we compare the different techniques proposed in the literature for estimating capacity and provide a practitioner perspective on their effectiveness. In particular, we study the performance of mutual information neural estimator (MINE), smoothed mutual information lower-bound estimator (SMILE), and directed information neural estimator (DINE) and provide insights on InfoNCE. We evaluated these algorithms in terms of their ability to learn the input distributions that are capacity approaching for the AWGN channel, the optical intensity channel, and peak power-constrained AWGN channel. For both scenarios, we provide insightful comments on various aspects of the training process, such as stability, sensitivity to initialization.