# Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models

arXiv: https://arxiv.org/abs/2605.15082 · HTML: https://mlsanity.com/paper/2605.15082/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 57 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 9 of 9)

- [On Learning Gaussian Multi-index Models with Gradient Flow](https://mlsanity.com/paper/2310.19793/) (2023)
- [Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions](https://mlsanity.com/paper/2405.15459/) (2024)
- [Phase Transitions for Feature Learning in Neural Networks](https://mlsanity.com/paper/2602.01434/) (2026)
- [Iteratively reweighted kernel machines efficiently learn sparse functions](https://mlsanity.com/paper/2505.08277/) (2025)
- [A general technique for approximating high-dimensional empirical kernel matrices](https://mlsanity.com/paper/2511.03892/) (2025)
- [Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs](https://mlsanity.com/paper/2406.02157/) (2024)
- [Learning Multi-Index Models with Hyper-Kernel Ridge Regression](https://mlsanity.com/paper/2510.02532/) (2025)
- [A Compositional Kernel Model for Feature Learning](https://mlsanity.com/paper/2509.14158/) (2025)
- [Local EGOP for Continuous Index Learning](https://mlsanity.com/paper/2601.07061/) (2026)

## Built on by (most cited first, 1 of 1)

- [Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines](https://mlsanity.com/paper/2610.09196/) (2026)

## Abstract (arXiv)

We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction. Specifically, we consider the problem of recovering a multi-index polynomial f^*(x)=h(Ux), with U∈ℝr× d and rll d, from finitely many data/label pairs. Importantly, the target function depends on input x only through the projection onto an unknown r-dimensional central subspace. The algorithm we analyze is appealingly simple: fit kernel ridge regression (KRR) to the data and compute the Average Gradient Outer Product (AGOP) from the fitted predictor. Our main results show that under reasonable assumptions the top r-dimensional eigenspace of AGOP provably recovers the central subspace, even in regimes when the prediction error remains large. Specifically, if the target function f^* has degree p^*, it is known that nasymp dp^* samples are necessary for KRR to achieve accurate prediction. In contrast, we show that if a low degree p component of f^* already carries all relevant directions for prediction, subspace recovery occurs in the much lower sample regime nasymp dp+δ for any δ∈(0,1). Our results thus demonstrate a separation between prediction and representation, and provide an explanation for why iterative kernel methods such as Recursive Feature Machines (RFM) can be sample-efficient in practice.