# How does training shape the Riemannian geometry of neural network representations?

arXiv: https://arxiv.org/abs/2301.11375 · HTML: https://mlsanity.com/paper/2301.11375/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2023.

## Built on by (most cited first, 5 of 5)

- [Learning curves for deep structured Gaussian feature models](https://mlsanity.com/paper/2303.00564/) (2023)
- [Compositional Generalization via Forced Rendering of Disentangled Latents](https://mlsanity.com/paper/2501.18797/) (2025)
- [Stimulus symmetries can confound representational similarity analyses](https://mlsanity.com/paper/2605.21324/) (2026)
- [Exploring Biologically Inspired Mechanisms of Adversarial Robustness](https://mlsanity.com/paper/2405.00679/) (2024)
- [Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers](https://mlsanity.com/paper/2609.39190/) (2026)

## Abstract (arXiv)

In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors. However, it is not always clear a priori what geometric constraints are appropriate for a given task. Here, we explore the possibility that one can uncover useful geometric inductive biases by studying how training molds the Riemannian geometry induced by unconstrained neural network feature maps. We first show that at infinite width, neural networks with random parameters induce highly symmetric metrics on input space. This symmetry is broken by feature learning: networks trained to perform classification tasks learn to magnify local areas along decision boundaries. This holds in deep networks trained on high-dimensional image classification tasks, and even in self-supervised representation learning. These results begin to elucidate how training shapes the geometry induced by unconstrained neural network feature maps, laying the groundwork for an understanding of this richly nonlinear form of feature learning.