# Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics

arXiv: https://arxiv.org/abs/1906.10720 · HTML: https://mlsanity.com/paper/1906.10720/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 30 of 39)

- [Auxiliary Tasks and Exploration Enable ObjectNav](https://mlsanity.com/paper/2104.04112/) (2021)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Analyzing And Editing Inner Mechanisms Of Backdoored Language Models](https://mlsanity.com/paper/2302.12461/) (2023)
- [Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems](https://mlsanity.com/paper/2111.01256/) (2021)
- [Bifurcations and loss jumps in RNN training](https://mlsanity.com/paper/2310.17561/) (2023)
- [An Operator Theoretic Approach for Analyzing Sequence Neural Networks](https://mlsanity.com/paper/2102.07824/) (2021)
- [Transformer Dynamics: A neuroscientific approach to interpretability of large language models](https://mlsanity.com/paper/2502.12131/) (2025)
- [Analyzing Populations of Neural Networks via Dynamical Model Embedding](https://mlsanity.com/paper/2302.14078/) (2023)
- [Expressive architectures enhance interpretability of dynamics-based neural population models](https://mlsanity.com/paper/2212.03771/) (2022)
- [Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs](https://mlsanity.com/paper/2506.07919/) (2025)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Warming up recurrent neural networks to maximise reachable multistability greatly improves learning](https://mlsanity.com/paper/2106.01001/) (2021)
- [Understanding Spending Behavior: Recurrent Neural Network Explanation and Interpretation](https://mlsanity.com/paper/2109.11871/) (2021)
- [Neural timescales from a computational perspective](https://mlsanity.com/paper/2409.02684/) (2024)
- [Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes](https://mlsanity.com/paper/2109.12434/) (2021)
- [Understanding How Encoder-Decoder Architectures Attend](https://mlsanity.com/paper/2110.15253/) (2021)
- [Trainability, Expressivity and Interpretability in Gated Neural ODEs](https://mlsanity.com/paper/2307.06398/) (2023)
- [Learning Dynamics of RNNs in Closed-Loop Environments](https://mlsanity.com/paper/2505.13567/) (2025)
- [Slow manifolds in recurrent networks encode working memory efficiently and robustly](https://mlsanity.com/paper/2101.03163/) (2021)
- [Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management](https://mlsanity.com/paper/2204.09218/) (2022)
- [From Data-Fitting to Discovery: Interpreting the Neural Dynamics of Motor Control through Reinforcement Learning](https://mlsanity.com/paper/2305.11107/) (2023)
- [Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning](https://mlsanity.com/paper/2307.12143/) (2023)
- [Interpretation of the Intent Detection Problem as Dynamics in a Low-dimensional Space](https://mlsanity.com/paper/2408.02838/) (2024)
- [A ghost mechanism: An analytical model of abrupt learning in recurrent networks](https://mlsanity.com/paper/2501.02378/) (2025)
- [Finding separatrices of dynamical flows with Deep Koopman Eigenfunctions](https://mlsanity.com/paper/2505.15231/) (2025)
- [Learning reveals invisible structure in low-rank RNNs](https://mlsanity.com/paper/2605.04115/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we would like to reverse engineer it--to obtain a quantitative, interpretable description of how it solves a particular task. Even for simple tasks, a detailed understanding of how recurrent networks work, or a prescription for how to develop such an understanding, remains elusive. In this work, we use tools from dynamical systems analysis to reverse engineer recurrent networks trained to perform sentiment classification, a foundational natural language processing task. Given a trained network, we find fixed points of the recurrent dynamics and linearize the nonlinear system around these fixed points. Despite their theoretical capacity to implement complex, high-dimensional computations, we find that trained networks converge to highly interpretable, low-dimensional representations. In particular, the topological structure of the fixed points and corresponding linearized dynamics reveal an approximate line attractor within the RNN, which we can use to quantitatively understand how the RNN solves the sentiment analysis task. Finally, we find this mechanism present across RNN architectures (including LSTMs, GRUs, and vanilla RNNs) trained on multiple datasets, suggesting that our findings are not unique to a particular architecture or dataset. Overall, these results demonstrate that surprisingly universal and human interpretable computations can arise across a range of recurrent networks.