# Capacity of Continuous Channels with Memory via Directed Information Neural Estimator

arXiv: https://arxiv.org/abs/2003.04179 · HTML: https://mlsanity.com/paper/2003.04179/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 5 of 5)

- [An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2103.06224/) (2021)
- [Neural Capacity Estimators: How Reliable Are They?](https://mlsanity.com/paper/2111.07401/) (2021)
- [Feedback Capacity of Parallel ACGN Channels and Kalman Filter: Power Allocation with Feedback](https://mlsanity.com/paper/2102.02730/) (2021)
- [A Perspective on Neural Capacity Estimation: Viability and Reliability](https://mlsanity.com/paper/2203.11793/) (2022)
- [Data-Driven Estimation of Capacity Upper Bounds](https://mlsanity.com/paper/2205.06471/) (2022)

## Abstract (arXiv)

Calculating the capacity (with or without feedback) of channels with memory and continuous alphabets is a challenging task. It requires optimizing the directed information (DI) rate over all channel input distributions. The objective is a multi-letter expression, whose analytic solution is only known for a few specific cases. When no analytic solution is present or the channel model is unknown, there is no unified framework for calculating or even approximating capacity. This work proposes a novel capacity estimation algorithm that treats the channel as a `black-box', both when feedback is or is not present. The algorithm has two main ingredients: (i) a neural distribution transformer (NDT) model that shapes a noise variable into the channel input distribution, which we are able to sample, and (ii) the DI neural estimator (DINE) that estimates the communication rate of the current NDT model. These models are trained by an alternating maximization procedure to both estimate the channel capacity and obtain an NDT for the optimal input distribution. The method is demonstrated on the moving average additive Gaussian noise channel, where it is shown that both the capacity and feedback capacity are estimated without knowledge of the channel transition kernel. The proposed estimation framework opens the door to a myriad of capacity approximation results for continuous alphabet channels that were inaccessible until now.