# The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold

arXiv: https://arxiv.org/abs/2511.01938 · HTML: https://mlsanity.com/paper/2511.01938/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 2 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024)
- [A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation](https://mlsanity.com/paper/2505.20172/) (2025)
- [When Representations Align: Universality in Representation Learning Dynamics](https://mlsanity.com/paper/2402.09142/) (2024)
- [Clustering and Alignment: Understanding the Training Dynamics in Modular Addition](https://mlsanity.com/paper/2408.09414/) (2024)

## Built on by (most cited first, 9 of 9)

- [The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology](https://mlsanity.com/paper/2603.05228/) (2026)
- [Distributional Spectral Diagnostics for Localizing Grokking Transitions](https://mlsanity.com/paper/2605.08237/) (2026)
- [Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics](https://mlsanity.com/paper/2605.20441/) (2026)
- [Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold](https://mlsanity.com/paper/2603.15492/) (2026)
- [Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets](https://mlsanity.com/paper/2605.15787/) (2026)
- [First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation](https://mlsanity.com/paper/2605.18845/) (2026)
- [What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy](https://mlsanity.com/paper/2606.18465/) (2026)
- [Fast Generalization after Interpolation via Critically Damped Momentum Optimization](https://mlsanity.com/paper/2606.01521/) (2026)
- [Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?](https://mlsanity.com/paper/2608.14803/) (2026)

## Abstract (arXiv)

Grokking is a puzzling phenomenon in neural networks where full generalization occurs only after a substantial delay following the complete memorization of the training data. Previous research has linked this delayed generalization to representation learning driven by weight decay, but the precise underlying dynamics remain elusive. In this paper, we argue that post-memorization learning can be understood through the lens of constrained optimization: gradient descent effectively minimizes the weight norm on the zero-loss manifold. We formally prove this in the limit of infinitesimally small learning rates and weight decay coefficients. To further dissect this regime, we introduce an approximation that decouples the learning dynamics of a subset of parameters from the rest of the network. Applying this framework, we derive a closed-form expression for the post-memorization dynamics of the first layer in a two-layer network. Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.