Dimensional Criticality at Grokking Across MLPs and Transformers
Not on the map. Why?
Abstract (arXiv)
Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems. Grokking in deep neural networks provides a striking example -- an abrupt transition from memorization to generalization long after training accuracy saturates -- yet robust macroscopic signatures of this transition remain elusive. Here we introduce TDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe that converts gradient snapshots into cascade statistics and extracts a macroscopic observable -- the time-resolved effective cascade dimension D(t) -- via grokking-aligned finite-size scaling. Across Transformers trained on modular addition and MLPs trained on XOR, we discover a localized dynamical crossing of the Gaussian diffusion baseline D=1 precisely at the generalization transition. The crossing direction is task-dependent: modular addition descends through D=1 (approaching from D>1), while XOR ascends (from D<1). This opposite-direction convergence is consistent with attraction toward a candidate shared critical manifold, rather than trivial residence near D ≈ 1. Negative controls confirm this picture: ungrokked runs remain supercritical (D>1) and never enter the post-transition regime. In addition, avalanche distributions exhibit heavy tails and finite-size scaling consistent with the dimensional exponent extracted from D(t). Shadow-probe controls (αtrain=0) confirm that D(t) is non-invasive, and grokked trajectories diverge from ungrokked ones in D(t) some 100--200 epochs before the behavioral transition.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 0 of its 3 references that are in the map. First posted on arXiv 6 Apr 2026.
Builds on
0 of 3 references.
No references here yet.
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2
- 2026What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropycited by 1
- 2026Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiencycited by 0
- 2026What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy16 Jun 2026
- 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics19 May 2026
- 2026Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiency3 May 2026