# Compression Through Time: An Interplay of Separability, Stationarity and Capacity

arXiv: https://arxiv.org/abs/2610.16660 · HTML: https://mlsanity.com/paper/2610.16660/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Recurrent Batch Normalization](https://mlsanity.com/paper/1603.09025/) (2016)
- [Capacity and Trainability in Recurrent Neural Networks](https://mlsanity.com/paper/1611.09913/) (2016)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges](https://mlsanity.com/paper/2310.08358/) (2023)
- [On Predictive Information in RNNs](https://mlsanity.com/paper/1910.09578/) (2019)
- [Emergence of Functionally Differentiated Structures via Mutual Information Minimization in Recurrent Neural Networks](https://mlsanity.com/paper/2507.12858/) (2025)
- [Identifying Information-Transfer Nodes in a Recurrent Neural Network Reveals Dynamic Representations](https://mlsanity.com/paper/2510.01271/) (2025)

## Abstract (arXiv)

Information-theoretic and geometric compression have been widely studied as possible explanations for generalization. Yet their temporal dynamics in recurrent networks remain largely unexplored. Using discretized LSTMs, whose finite embedding space enables direct estimates of information quantities, we observe pronounced compression across the final timesteps of many datasets. We explain this effect through three competing mechanisms: (i) class separability permits stronger compression at the final timestep; (ii) shared recurrent dynamics propagate this pressure across timesteps with similar activation states, making its extent depend on temporal stationarity; and (iii) limited representational capacity opposes prolonged compression because it restricts information available for subsequent computation. We systematically test these mechanisms through controlled synthetic interventions and real-world sequence tasks. Our results identify class separability, temporal stationarity, and network capacity as key determinants of temporal compression, and suggest that compression acts as a regularizer-potentially beneficial when excess capacity is available, but harmful to generalization when it consumes memory needed for the task.