# Universality and individuality in neural dynamics across large populations of recurrent networks

arXiv: https://arxiv.org/abs/1907.08549 · HTML: https://mlsanity.com/paper/1907.08549/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2019.

## Built on by (most cited first, 30 of 61)

- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Generalized Shape Metrics on Neural Representations](https://mlsanity.com/paper/2110.14739/) (2021)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis](https://mlsanity.com/paper/2306.10168/) (2023)
- [Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems](https://mlsanity.com/paper/2111.01256/) (2021)
- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Representational dissimilarity metric spaces for stochastic neural networks](https://mlsanity.com/paper/2211.11665/) (2022)
- [Duality of Bures and Shape Distances with Implications for Comparing Neural Representations](https://mlsanity.com/paper/2311.11436/) (2023)
- [Interpretable statistical representations of neural population dynamics and geometry](https://mlsanity.com/paper/2304.03376/) (2023)
- [Low Tensor Rank Learning of Neural Dynamics](https://mlsanity.com/paper/2308.11567/) (2023)
- [On the Origins of the Block Structure Phenomenon in Neural Network Representations](https://mlsanity.com/paper/2202.07184/) (2022)
- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [When Representations Align: Universality in Representation Learning Dynamics](https://mlsanity.com/paper/2402.09142/) (2024)
- [Formation of Representations in Neural Networks](https://mlsanity.com/paper/2410.03006/) (2024)
- [Meta-Dynamical State Space Models for Integrative Neural Data Analysis](https://mlsanity.com/paper/2410.05454/) (2024)
- [Sources of Richness and Ineffability for Phenomenally Conscious States](https://mlsanity.com/paper/2302.06403/) (2023)
- [Analyzing Populations of Neural Networks via Dynamical Model Embedding](https://mlsanity.com/paper/2302.14078/) (2023)
- [Deep Recurrent Encoder: A scalable end-to-end network to model brain signals](https://mlsanity.com/paper/2103.02339/) (2021)
- [Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limit](https://mlsanity.com/paper/2112.09305/) (2021)
- [Expressive architectures enhance interpretability of dynamics-based neural population models](https://mlsanity.com/paper/2212.03771/) (2022)
- [InputDSA: Demixing then Comparing Recurrent and Externally Driven Dynamics](https://mlsanity.com/paper/2510.25943/) (2025)
- [FONDUE: an algorithm to find the optimal dimensionality of the latent representations of variational autoencoders](https://mlsanity.com/paper/2209.12806/) (2022)
- [Deconstructing deep active inference](https://mlsanity.com/paper/2303.01618/) (2023)
- [Efficient and robust multi-task learning in the brain with modular latent primitives](https://mlsanity.com/paper/2105.14108/) (2021)
- [Geometric and Topological Inference for Deep Representations of Complex Networks](https://mlsanity.com/paper/2203.05488/) (2022)
- [Estimating Shape Distances on Neural Representations with Limited Samples](https://mlsanity.com/paper/2310.05742/) (2023)
- [Lyapunov-Guided Representation of Recurrent Neural Network Performance](https://mlsanity.com/paper/2204.04876/) (2022)

## Abstract (arXiv)

Task-based modeling with recurrent neural networks (RNNs) has emerged as a popular way to infer the computational function of different brain regions. These models are quantitatively assessed by comparing the low-dimensional neural representations of the model with the brain, for example using canonical correlation analysis (CCA). However, the nature of the detailed neurobiological inferences one can draw from such efforts remains elusive. For example, to what extent does training neural networks to solve common tasks uniquely determine the network dynamics, independent of modeling architectural choices? Or alternatively, are the learned dynamics highly sensitive to different model choices? Knowing the answer to these questions has strong implications for whether and how we should use task-based RNN modeling to understand brain dynamics. To address these foundational questions, we study populations of thousands of networks, with commonly used RNN architectures, trained to solve neuroscientifically motivated tasks and characterize their nonlinear dynamics. We find the geometry of the RNN representations can be highly sensitive to different network architectures, yielding a cautionary tale for measures of similarity that rely representational geometry, such as CCA. Moreover, we find that while the geometry of neural dynamics can vary greatly across architectures, the underlying computational scaffold---the topological structure of fixed points, transitions between them, limit cycles, and linearized dynamics---often appears universal across all architectures.