# Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer

arXiv: https://arxiv.org/abs/2211.16607 · HTML: https://mlsanity.com/paper/2211.16607/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [Learnability for the Information Bottleneck](https://mlsanity.com/paper/1907.07331/) (2019)
- [Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck](https://mlsanity.com/paper/2106.01177/) (2021)
- [Large-scale nonlinear Granger causality: A data-driven, multivariate approach to recovering directed networks from short time-series data](https://mlsanity.com/paper/2009.04681/) (2020)
- [ITENE: Intrinsic Transfer Entropy Neural Estimator](https://mlsanity.com/paper/1912.07277/) (2019)
- [Forecasting and Granger Modelling with Non-linear Dynamical Dependencies](https://mlsanity.com/paper/1706.08811/) (2017)

## Built on by (most cited first, 3 of 3)

- [Which bits went where? Past and future transfer entropy decomposition with the information bottleneck](https://mlsanity.com/paper/2411.04992/) (2024)
- [InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis](https://mlsanity.com/paper/2506.08884/) (2025)
- [Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions](https://mlsanity.com/paper/2610.01175/) (2026)

## Abstract (arXiv)

When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream). For example, fluctuations in temperature at a weather station can be predicted using both temperatures and barometric readings. However, a challenge when modelling such data is that it is easy for a neural network to rely on the greatest joint correlations within the target stream, which may ignore a crucial but small information transfer from the source to the target stream. As well, there are often situations where the target stream may have previously been modelled independently and it would be useful to use that model to inform a new joint model. Here, we develop an information bottleneck approach for conditional learning on two dependent streams of data. Our method, which we call Transfer Entropy Bottleneck (TEB), allows one to learn a model that bottlenecks the directed information transferred from the source variable to the target variable, while quantifying this information transfer within the model. As such, TEB provides a useful new information bottleneck approach for modelling two statistically dependent streams of data in order to make predictions about one of them.