# Grokfast: Accelerated Grokking by Amplifying Slow Gradients

arXiv: https://arxiv.org/abs/2405.20233 · HTML: https://mlsanity.com/paper/2405.20233/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023)
- [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition](https://mlsanity.com/paper/2402.15175/) (2024)
- [Training Dynamics of Deep Network Linear Regions](https://mlsanity.com/paper/2310.12977/) (2023)

## Built on by (most cited first, 30 of 41)

- [The AdEMAMix Optimizer: Better, Faster, Older](https://mlsanity.com/paper/2409.03137/) (2024)
- [WARP: On the Benefits of Weight Averaged Rewarded Policies](https://mlsanity.com/paper/2406.16768/) (2024)
- [RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?](https://mlsanity.com/paper/2509.21016/) (2025)
- [Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model](https://mlsanity.com/paper/2504.13292/) (2025)
- [Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers](https://mlsanity.com/paper/2504.20752/) (2025)
- [Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)](https://mlsanity.com/paper/2502.21009/) (2025)
- [Mechanistic Insights into Grokking from the Embedding Layer](https://mlsanity.com/paper/2505.15624/) (2025)
- [A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization](https://mlsanity.com/paper/2603.25009/) (2026)
- [Language Models "Grok" to Copy](https://mlsanity.com/paper/2409.09281/) (2024)
- [An Ensemble Approach for Brain Tumor Segmentation and Synthesis](https://mlsanity.com/paper/2411.17617/) (2024)
- [The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology](https://mlsanity.com/paper/2603.05228/) (2026)
- [Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design](https://mlsanity.com/paper/2605.15871/) (2026)
- [Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds](https://mlsanity.com/paper/2605.09724/) (2026)
- [Grokking as a Falsifiable Finite-Size Transition](https://mlsanity.com/paper/2603.24746/) (2026)
- [Structure-Specific Representational Priors Causally Control the Grokking Delay](https://mlsanity.com/paper/2607.04333/) (2026)
- [NeuralGrok: Accelerate Grokking by Neural Gradient Transformation](https://mlsanity.com/paper/2504.17243/) (2025)
- [GrokAlign: Geometric Characterisation and Acceleration of Grokking](https://mlsanity.com/paper/2506.12284/) (2025)
- [Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking](https://mlsanity.com/paper/2510.04930/) (2025)
- [MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates](https://mlsanity.com/paper/2510.05361/) (2025)
- [Greedy Alignment Principle for Optimizer Selection](https://mlsanity.com/paper/2512.06370/) (2025)
- [The Order Is The Message](https://mlsanity.com/paper/2603.25047/) (2026)
- [Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin](https://mlsanity.com/paper/2604.13123/) (2026)
- [ILDR: Geometric Early Detection of Grokking](https://mlsanity.com/paper/2604.20923/) (2026)
- [Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokking](https://mlsanity.com/paper/2606.12966/) (2026)
- [Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking](https://mlsanity.com/paper/2504.03162/) (2025)
- [When Data Falls Short: Grokking Below the Critical Threshold](https://mlsanity.com/paper/2511.04760/) (2025)
- [Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers](https://mlsanity.com/paper/2601.09049/) (2026)
- [Dynamic Momentum Recalibration in Online Gradient Learning](https://mlsanity.com/paper/2603.06120/) (2026)
- [Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory](https://mlsanity.com/paper/2605.12394/) (2026)
- [Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization](https://mlsanity.com/paper/2606.32000/) (2026)

## Abstract (arXiv)

One puzzling artifact in machine learning dubbed grokking is where delayed generalization is achieved tenfolds of iterations after near perfect overfitting to the training data. Focusing on the long delay itself on behalf of machine learning practitioners, our goal is to accelerate generalization of a model under grokking phenomenon. By regarding a series of gradients of a parameter over training iterations as a random signal over time, we can spectrally decompose the parameter trajectories under gradient descent into two components: the fast-varying, overfitting-yielding component and the slow-varying, generalization-inducing component. This analysis allows us to accelerate the grokking phenomenon more than × 50 with only a few lines of code that amplifies the slow-varying components of gradients. The experiments show that our algorithm applies to diverse tasks involving images, languages, and graphs, enabling practical availability of this peculiar artifact of sudden generalization. Our code is available at https://github.com/ironjr/grokfast.