mlsanityData to 14 Oct 2026

What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy

arXiv 2606.18465 ↗First posted 16 Jun 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Grokking, the delayed jump from memorization to generalization, is usually tied to the weight norm: a smaller norm generalizes sooner. We ask what the norm actually controls. Holding the weight norm fixed by clamping and varying only an output temperature, we slide the grokking delay across its entire norm-induced range under cross-entropy; matching the effective logit scale back to baseline recovers about 85% of the delay at two moduli. Across a grid of norms and temperatures the delay collapses onto the logit scale alone (R2 = 0.97), with the norm adding 1-2% beyond it. The effect is loss-dependent: under mean-squared error the logit scale is pinned and the norm acts through a different route. A memorization control, a float64 softmax-collapse audit, and a no-LayerNorm transformer point to the same channel. Forking arms from one identical state, the delay follows the held norm value and not the clamp operation, which closes a rescaling-artifact concern. The proximal variable is the logit scale and the softmax saturation it drives; the weight norm is only an upstream handle. All numbers, tables, and figures reproduce from released code and data.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 21 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

20 of 21 references.

References · 20

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  6. 2023Explaining grokking through circuit efficiencycited by 111
  7. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  8. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  9. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  10. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  11. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  12. 2024Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transitioncited by 17
8 more
  1. 2025Grokking Beyond the Euclidean Norm of Model Parameterscited by 15
  2. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
  3. 2025The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifoldcited by 9
  4. 2026Grokking as Dimensional Phase Transition in Neural Networkscited by 4
  5. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  6. 2026Dimensional Criticality at Grokking Across MLPs and Transformerscited by 3
  7. 2025Predator-Prey Model: Driven Hunt for Accelerated Grokkingcited by 1
  8. 2026The Implicit Bias of Logit Regularizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.