# Embedding of Low-Dimensional Sensory Dynamics in Recurrent Networks: Implications for the Geometry of Neural Representation

arXiv: https://arxiv.org/abs/2601.19019 · HTML: https://mlsanity.com/paper/2601.19019/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 90 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Stable Recurrent Models](https://mlsanity.com/paper/1805.10369/) (2018)
- [Attractor and integrator networks in the brain](https://mlsanity.com/paper/2112.03978/) (2021)
- [Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity](https://mlsanity.com/paper/2107.04084/) (2021)
- [On the stability properties of Gated Recurrent Units neural networks](https://mlsanity.com/paper/2011.06806/) (2020)
- [Embedding Theory of Reservoir Computing and Reducing Reservoir Network Using Time Delays](https://mlsanity.com/paper/2303.09042/) (2023)
- [Modeling and Contractivity of Neural-Synaptic Networks with Hebbian Learning](https://mlsanity.com/paper/2204.05382/) (2022)
- [Delay Embedding Theory of Neural Sequence Models](https://mlsanity.com/paper/2406.11993/) (2024)
- [Euclidean Contractivity of Neural Networks with Symmetric Weights](https://mlsanity.com/paper/2302.13452/) (2023)

## Abstract (arXiv)

Neural population activity in sensory cortex is organized on low-dimensional manifolds, but why such manifolds arise and what determines their geometry remain unclear. We model cortical populations as recurrent circuits driven by low-dimensional regular sensory dynamics (circles, tori). Combining generalized synchronization and delay-embedding theory, we show that contracting recurrent networks generically develop smooth internal manifolds embedding the sensory dynamics. The dimensional requirement is modest: N>2d suffices, where d is the intrinsic sensory dimension (compatible with Whitney and Takens bounds). We prove a prediction-separation result linking representational geometry to predictive performance without assuming contraction: accurate prediction forces state separation up to a resolution set by prediction error, yielding categorical boundaries, metameric equivalence, and discrimination thresholds. Numerical experiments with trained tanh RNNs recover ring- and torus-shaped hidden manifolds; state separation improves sharply at the 2d+1 threshold. Training pushes networks beyond strict contraction, yet embedding persists, indicating sufficient but not necessary conditions. These results provide a mechanistic account of why sensory manifolds emerge in recurrent circuits and how prediction constrains their resolution.