# Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines

arXiv: https://arxiv.org/abs/2610.09196 · HTML: https://mlsanity.com/paper/2610.09196/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 45 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 8 of 8)

- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Implicit Bias of Linear Equivariant Networks](https://mlsanity.com/paper/2110.06084/) (2021)
- [Iteratively reweighted kernel machines efficiently learn sparse functions](https://mlsanity.com/paper/2505.08277/) (2025)
- [Efficient and accurate steering of Large Language Models through attention-guided feature learning](https://mlsanity.com/paper/2602.00333/) (2026)
- [Contextual Linear Activation Steering of Language Models](https://mlsanity.com/paper/2604.24693/) (2026)
- [The Expected Jacobian Outerproduct: Theory and Empirics](https://mlsanity.com/paper/2006.03550/) (2020)
- [Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization](https://mlsanity.com/paper/2502.01594/) (2025)
- [Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models](https://mlsanity.com/paper/2605.15082/) (2026)

## Abstract (arXiv)

Recursive feature machines (RFMs) learn representations of data by alternating between fitting a predictor to a dataset and updating features of that predictor using the average gradient outer product (AGOP). Connections between AGOPs and feature learning in neural networks motivate linear RFMs as a simple setting for analyzing how representations evolve during training. Here, we study the dynamics and statistics of linear RFM in noisy multi-output regression with isotropic sub-Gaussian input data and targets generated by a low-rank teacher matrix of dimension d. We extend the known connection between linear RFM and iteratively reweighted least squares from the interpolating setting to ridge-regularized multi-output regression with noise. We show that the learned feature matrix remains close to its infinite-data ideal counterpart at every iteration. Namely, for n samples, we show the error in the feature matrix decays as O(√d/n) with high probability. Experiments on real-world text and single-cell gene-expression data illustrate the features learned by this simple linear model.