# Usable Information and Evolution of Optimal Representations During Training

arXiv: https://arxiv.org/abs/2010.02459 · HTML: https://mlsanity.com/paper/2010.02459/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Critical Learning Periods for Multisensory Integration in Deep Networks](https://mlsanity.com/paper/2210.04643/) (2022)
- [Gacs-Korner Common Information Variational Autoencoder](https://mlsanity.com/paper/2205.12239/) (2022)
- [A Measure of the Complexity of Neural Representations based on Partial Information Decomposition](https://mlsanity.com/paper/2209.10438/) (2022)
- [Analyzing Local Representations of Self-supervised Vision Transformers](https://mlsanity.com/paper/2401.00463/) (2023)
- [Does YOLO Really Need to See Every Training Image in Every Epoch?](https://mlsanity.com/paper/2603.17684/) (2026)
- [Graph Representation Learning via Aggregation Enhancement](https://mlsanity.com/paper/2201.12843/) (2022)

## Abstract (arXiv)

We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training. We show that the implicit regularization coming from training with Stochastic Gradient Descent with a high learning-rate and small batch size plays an important role in learning minimal sufficient representations for the task. In the process of arriving at a minimal sufficient representation, we find that the content of the representation changes dynamically during training. In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training, before being later discarded. In addition, we evaluate how perturbing the initial part of training impacts the learning dynamics and the resulting representations. We show these effects on both perceptual decision-making tasks inspired by neuroscience literature, as well as on standard image classification tasks.