# Measuring Information Transfer in Neural Networks

arXiv: https://arxiv.org/abs/2009.07624 · HTML: https://mlsanity.com/paper/2009.07624/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2020.

## Built on by (most cited first, 4 of 4)

- [In-context learning and Occam's razor](https://mlsanity.com/paper/2410.14086/) (2024)
- [Feature Forgetting in Continual Representation Learning](https://mlsanity.com/paper/2205.13359/) (2022)
- [Compute-Optimal LLMs Provably Generalize Better With Scale](https://mlsanity.com/paper/2504.15208/) (2025)
- [Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data](https://mlsanity.com/paper/2607.11883/) (2026)

## Abstract (arXiv)

Quantifying the information content in a neural network model is essentially estimating the model's Kolmogorov complexity. Recent success of prequential coding on neural networks points to a promising path of deriving an efficient description length of a model. We propose a practical measure of the generalizable information in a neural network model based on prequential coding, which we term Information Transfer (LIT). Theoretically, LIT is an estimation of the generalizable part of a model's information content. In experiments, we show that LIT is consistently correlated with generalizable information and can be used as a measure of patterns or "knowledge" in a model or a dataset. Consequently, LIT can serve as a useful analysis tool in deep learning. In this paper, we apply LIT to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. LIT provides an information perspective which helps us discover new insights into neural network learning.