# Interpreting recurrent neural networks behaviour via excitable network attractors

arXiv: https://arxiv.org/abs/1807.10478 · HTML: https://mlsanity.com/paper/1807.10478/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2018.

## Built on by (most cited first, 15 of 15)

- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Seeing double with a multifunctional reservoir computer](https://mlsanity.com/paper/2305.05799/) (2023)
- [Recent Advances in Malware Detection: Graph Learning and Explainability](https://mlsanity.com/paper/2502.10556/) (2025)
- [Outdoor Monocular Depth Estimation: A Research Review](https://mlsanity.com/paper/2205.01399/) (2022)
- [Confabulation dynamics in a reservoir computer: Filling in the gaps with untrained attractors](https://mlsanity.com/paper/2505.04792/) (2025)
- [Warming up recurrent neural networks to maximise reachable multistability greatly improves learning](https://mlsanity.com/paper/2106.01001/) (2021)
- [Understanding Spending Behavior: Recurrent Neural Network Explanation and Interpretation](https://mlsanity.com/paper/2109.11871/) (2021)
- [Scope and Sense of Explainability for AI-Systems](https://mlsanity.com/paper/2112.10551/) (2021)
- [Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management](https://mlsanity.com/paper/2204.09218/) (2022)
- [Transitions in echo index and dependence on input repetitions](https://mlsanity.com/paper/2309.04728/) (2023)
- [Interpretation of the Intent Detection Problem as Dynamics in a Low-dimensional Space](https://mlsanity.com/paper/2408.02838/) (2024)
- [On the dimension of pullback attractors in recurrent neural networks](https://mlsanity.com/paper/2501.11357/) (2025)
- [Interpretability of the Intent Detection Problem: A New Approach](https://mlsanity.com/paper/2601.17156/) (2026)
- [Markovian Circuit Tracing for Transformer State Dynamic](https://mlsanity.com/paper/2605.20824/) (2026)
- [Basin Geometry and Reliable Recall of Dynamical Memories in Reservoir Computing](https://mlsanity.com/paper/2609.01914/) (2026)

## Abstract (arXiv)

Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.