# Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2605.12763 · HTML: https://mlsanity.com/paper/2605.12763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 30 of 31)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Tensor Programs II: Neural Tangent Kernel for Any Architecture](https://mlsanity.com/paper/2006.14548/) (2020)
- [The Recurrent Neural Tangent Kernel](https://mlsanity.com/paper/2006.10246/) (2020)
- [Hutch++: Optimal Stochastic Trace Estimation](https://mlsanity.com/paper/2010.09649/) (2020)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Linking connectivity, dynamics and computations in low-rank recurrent neural networks](https://mlsanity.com/paper/1711.09672/) (2017)
- [From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks](https://mlsanity.com/paper/2409.14623/) (2024)
- [Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks](https://mlsanity.com/paper/2304.03408/) (2023)
- [A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks](https://mlsanity.com/paper/1907.02649/) (2019)
- [SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient](https://mlsanity.com/paper/1811.04504/) (2018)
- [A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2206.12543/) (2022)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [A Theory of Neural Tangent Kernel Alignment and Its Influence on Training](https://mlsanity.com/paper/2105.14301/) (2021)
- [Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks](https://mlsanity.com/paper/2010.00879/) (2020)
- [The interplay between randomness and structure during learning in RNNs](https://mlsanity.com/paper/2006.11036/) (2020)
- [Bifurcations and loss jumps in RNN training](https://mlsanity.com/paper/2310.17561/) (2023)
- [How connectivity structure shapes rich and lazy learning in neural circuits](https://mlsanity.com/paper/2310.08513/) (2023)
- [Low Tensor Rank Learning of Neural Dynamics](https://mlsanity.com/paper/2308.11567/) (2023)
- [Sketchy Empirical Natural Gradient Methods for Deep Learning](https://mlsanity.com/paper/2006.05924/) (2020)
- [On the Neural Tangent Kernel of Equilibrium Models](https://mlsanity.com/paper/2310.14062/) (2023)
- [Connectivity structure and dynamics of nonlinear recurrent neural networks](https://mlsanity.com/paper/2409.01969/) (2024)
- [Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation](https://mlsanity.com/paper/2511.10796/) (2025)
- [Memorisation and forgetting in a learning Hopfield neural network: bifurcation mechanisms, attractors and basins](https://mlsanity.com/paper/2508.10765/) (2025)

## Abstract (arXiv)

Rich learning in recurrent neural networks often proceeds through sudden transitions in latent dynamics, but there is little theory predicting how gradient descent behaves during these events. We study the local learning geometry near codimension-one bifurcations through the global empirical Neural Tangent Kernel (GeNTK). Under local center-manifold conditions, and when bifurcation-related sensitivity dominates bounded residual terms, we show that the global parameter-to-state Jacobian (Dθ h) is approximated by a low-rank normal-form operator. The induced GeNTK and Fisher information matrix therefore become strongly amplified and anisotropic, concentrating toward a rank-one channel for the four scalar codimension-one bifurcations and a rank-two real channel for a Neimark--Sacker bifurcation. Controlled high-dimensional RNN experiments validate this operator reduction. In learned RNNs, the same low-rank concentration coincides with abrupt loss changes and subtask interference, while a local projection predicts the sign of these effects near isolated events. Finally, in an input-driven 15-task LeakyRNN, GeNTK amplification aligns with continuation-detected changes in the MemoryPro dynamics. These results suggest a tractable operator-level description of learning near dynamical transitions, together with scalable diagnostics for amplified low-dimensional learning geometry.