# Topological Invariance and Breakdown in Learning

arXiv: https://arxiv.org/abs/2510.02670 · HTML: https://mlsanity.com/paper/2510.02670/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Topology of deep neural networks](https://mlsanity.com/paper/2004.06093/) (2020)
- [A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation](https://mlsanity.com/paper/1810.13243/) (2018)
- [Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks](https://mlsanity.com/paper/2111.13171/) (2021)
- [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019)
- [Why Warmup the Learning Rate? Underlying Mechanisms and Improvements](https://mlsanity.com/paper/2406.09405/) (2024)
- [Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks](https://mlsanity.com/paper/2306.04251/) (2023)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width](https://mlsanity.com/paper/2302.12250/) (2023)
- [Experimental Observations of the Topology of Convolutional Neural Network Activations](https://mlsanity.com/paper/2212.00222/) (2022)
- [Remove Symmetries to Control Model Expressivity and Improve Optimization](https://mlsanity.com/paper/2408.15495/) (2024)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [New Evidence of the Two-Phase Learning Dynamics of Neural Networks](https://mlsanity.com/paper/2505.13900/) (2025)
- [A universal compression theory for lottery ticket hypothesis and neural scaling laws](https://mlsanity.com/paper/2510.00504/) (2025)

## Built on by (most cited first, 1 of 1)

- [On the Epistemic Uncertainty of Overparametrized Neural Networks](https://mlsanity.com/paper/2605.25234/) (2026)

## Abstract (arXiv)

We prove that for a broad class of permutation-equivariant learning rules (including SGD, Adam, and others), the training process induces a bi-Lipschitz mapping between neurons and strongly constrains the topology of the neuron distribution during training. This result reveals a qualitative difference between small and large learning rates η. With a learning rate below a topological critical point η^*, the training is constrained to preserve all topological structure of the neurons. In contrast, above η^*, the learning process allows for topological simplification, making the neuron manifold progressively coarser and thereby reducing the model's expressivity. Viewed in combination with the recent discovery of the edge of stability phenomenon, the learning dynamics of neuron networks under gradient descent can be divided into two phases: first they undergo smooth optimization under topological constraints, and then enter a second phase where they learn through drastic topological simplifications. A key feature of our theory is that it is independent of specific architectures or loss functions, enabling the universal application of topological methods to the study of deep learning.