# A Theory of Feature Learning in Kernel Models

arXiv: https://arxiv.org/abs/2310.11736 · HTML: https://mlsanity.com/paper/2310.11736/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 36 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Built on by (most cited first, 3 of 3)

- [Average gradient outer product as a mechanism for deep neural collapse](https://mlsanity.com/paper/2402.13728/) (2024)
- [Improving Adaptivity via Over-Parameterization in Sequence Models](https://mlsanity.com/paper/2409.00894/) (2024)
- [On the Limitation of Kernel Dependence Maximization for Feature Selection](https://mlsanity.com/paper/2406.06903/) (2024)

## Abstract (arXiv)

We study feature learning in a compositional variant of kernel ridge regression in which the predictor is applied to a learnable linear transformation of the input. When the response depends on the input only through a low-dimensional predictive subspace, we show that all global minimizers of the population objective for the linear transformation annihilate directions orthogonal to this subspace, and in certain regimes, exactly identify the subspace. Moreover, we show that global minimizers of the finite-sample objective inherit the exact same low-dimensional structure with high probability, even without any explicit penalization on the linear transformation.