# Aligned and oblique dynamics in recurrent neural networks

arXiv: https://arxiv.org/abs/2307.07654 · HTML: https://mlsanity.com/paper/2307.07654/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2023.

## Built on by (most cited first, 10 of 10)

- [How connectivity structure shapes rich and lazy learning in neural circuits](https://mlsanity.com/paper/2310.08513/) (2023)
- [Fading memory as inductive bias in residual recurrent networks](https://mlsanity.com/paper/2307.14823/) (2023)
- [Learning Dynamics of RNNs in Closed-Loop Environments](https://mlsanity.com/paper/2505.13567/) (2025)
- [A unified theory of feature learning in RNNs and DNNs](https://mlsanity.com/paper/2602.15593/) (2026)
- [Reasoning Models Don't Just Think Longer, They Move Differently](https://mlsanity.com/paper/2605.15454/) (2026)
- [Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example](https://mlsanity.com/paper/2506.06904/) (2025)
- [Algorithm Development in Neural Networks: Insights from the Streaming Parity Task](https://mlsanity.com/paper/2507.09897/) (2025)
- [JEDI: Jointly Embedded Inference of Neural Dynamics](https://mlsanity.com/paper/2603.10489/) (2026)
- [Factorization Regret mediates compositional generalization in latent space](https://mlsanity.com/paper/2603.27134/) (2026)
- [Task-Restricted Symmetries in Recurrent Weight Space](https://mlsanity.com/paper/2606.18457/) (2026)

## Abstract (arXiv)

The relation between neural activity and behaviorally relevant variables is at the heart of neuroscience research. When strong, this relation is termed a neural representation. There is increasing evidence, however, for partial dissociations between activity in an area and relevant external variables. While many explanations have been proposed, a theoretical framework for the relationship between external and internal variables is lacking. Here, we utilize recurrent neural networks (RNNs) to explore the question of when and how neural dynamics and the network's output are related from a geometrical point of view. We find that training RNNs can lead to two dynamical regimes: dynamics can either be aligned with the directions that generate output variables, or oblique to them. We show that the choice of readout weight magnitude before training can serve as a control knob between the regimes, similar to recent findings in feedforward networks. These regimes are functionally distinct. Oblique networks are more heterogeneous and suppress noise in their output directions. They are furthermore more robust to perturbations along the output directions. Crucially, the oblique regime is specific to recurrent (but not feedforward) networks, arising from dynamical stability considerations. Finally, we show that tendencies towards the aligned or the oblique regime can be dissociated in neural recordings. Altogether, our results open a new perspective for interpreting neural activity by relating network dynamics and their output.