# Internal representation dynamics and geometry in recurrent neural networks

arXiv: https://arxiv.org/abs/2001.03255 · HTML: https://mlsanity.com/paper/2001.03255/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2020.

## Built on by (most cited first, 2 of 2)

- [Generalization in Neural Networks: A Broad Survey](https://mlsanity.com/paper/2209.01610/) (2022)
- [Using a Cross-Task Grid of Linear Probes to Interpret CNN Model Predictions On Retinal Images](https://mlsanity.com/paper/2107.11468/) (2021)

## Abstract (arXiv)

The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established. However, unlike deep, and convolution networks where we can attribute the recognition of a certain feature to every layer, it is unclear what "sub-task" a single recurrent step or layer accomplishes. Our work seeks to shed light onto how a vanilla RNN implements a simple classification task by analysing the dynamics of the network and the geometric properties of its hidden states. We find that early internal representations are evocative of the real labels of the data but this information is not directly accessible to the output layer. Furthermore the network's dynamics and the sequence length are both critical to correct classifications even when there is no additional task relevant information provided.