# A unified theory of feature learning in RNNs and DNNs

arXiv: https://arxiv.org/abs/2602.15593 · HTML: https://mlsanity.com/paper/2602.15593/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Neural Networks as Kernel Learners: The Silent Alignment Effect](https://mlsanity.com/paper/2111.00034/) (2021)
- [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022)
- [A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data](https://mlsanity.com/paper/2402.16991/) (2024)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [The Recurrent Neural Tangent Kernel](https://mlsanity.com/paper/2006.10246/) (2020)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Bayes-optimal Learning of Deep Random Networks of Extensive-width](https://mlsanity.com/paper/2302.00375/) (2023)
- [Unified field theoretical approach to deep and recurrent neuronal networks](https://mlsanity.com/paper/2112.05589/) (2021)
- [Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks](https://mlsanity.com/paper/2307.11807/) (2023)
- [Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer](https://mlsanity.com/paper/2502.02531/) (2025)
- [Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes](https://mlsanity.com/paper/2405.17580/) (2024)
- [Adaptive kernel predictors from feature-learning infinite limits of neural networks](https://mlsanity.com/paper/2502.07998/) (2025)
- [Transition to chaos in random neuronal networks](https://mlsanity.com/paper/1508.06486/) (2015)
- [Coding schemes in neural networks learning classification tasks](https://mlsanity.com/paper/2406.16689/) (2024)
- [Aligned and oblique dynamics in recurrent neural networks](https://mlsanity.com/paper/2307.07654/) (2023)
- [From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning](https://mlsanity.com/paper/2502.03210/) (2025)
- [Origami in N dimensions: How feed-forward networks manufacture linear separability](https://mlsanity.com/paper/2203.11355/) (2022)
- [Forgetting leads to chaos in attractor networks](https://mlsanity.com/paper/2112.00119/) (2021)
- [Intrinsically-generated fluctuating activity in excitatory-inhibitory networks](https://mlsanity.com/paper/1605.04221/) (2016)

## Built on by (most cited first, 2 of 2)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence](https://mlsanity.com/paper/2606.12058/) (2026)

## Abstract (arXiv)

Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning (μP) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.