# Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model arXiv: https://arxiv.org/abs/2504.13292 ยท HTML: https://mlsanity.com/paper/2504.13292/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 41 references that are in the map. - First posted on arXiv 17 Apr 2025. ## Builds on (most cited first, 30 of 33) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023) - [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022) - [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022) - [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023) - [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022) - [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022) - [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023) - [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023) - [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023) - [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023) - [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022) - [Grokking modular arithmetic](https://mlsanity.com/paper/2301.02679/) (2023) - [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023) - [A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks](https://mlsanity.com/paper/2303.11873/) (2023) - [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023) - [Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization](https://mlsanity.com/paper/2405.15071/) (2024) - [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024) - [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023) - [Grokfast: Accelerated Grokking by Amplifying Slow Gradients](https://mlsanity.com/paper/2405.20233/) (2024) - [Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks](https://mlsanity.com/paper/2406.02550/) (2024) - [Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition](https://mlsanity.com/paper/2407.12332/) (2024) - [Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data](https://mlsanity.com/paper/2310.02541/) (2023) - [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023) - [A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2206.12543/) (2022) - [Grokking phase transitions in learning local rules with gradient descent](https://mlsanity.com/paper/2210.15435/) (2022) - [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024) - [Critical Data Size of Language Models from a Grokking Perspective](https://mlsanity.com/paper/2401.10463/) (2024) ## Built on by (most cited first, 13 of 13) - [Defenses Against Prompt Attacks Learn Surface Heuristics](https://mlsanity.com/paper/2601.07185/) (2026) - [To Grok Grokking: Provable Grokking in Ridge Regression](https://mlsanity.com/paper/2601.19791/) (2026) - [Mechanistic Insights into Grokking from the Embedding Layer](https://mlsanity.com/paper/2505.15624/) (2025) - [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](https://mlsanity.com/paper/2605.06152/) (2026) - [GrokAlign: Geometric Characterisation and Acceleration of Grokking](https://mlsanity.com/paper/2506.12284/) (2025) - [When Data Falls Short: Grokking Below the Critical Threshold](https://mlsanity.com/paper/2511.04760/) (2025) - [Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers](https://mlsanity.com/paper/2601.09049/) (2026) - [Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction](https://mlsanity.com/paper/2606.05863/) (2026) - [Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization](https://mlsanity.com/paper/2606.32000/) (2026) - [How to Tame Grokking: Representation Geometry as a Control Signal](https://mlsanity.com/paper/2607.11666/) (2026) - [Tracing the Path to Grokking: Embeddings, Dropout, and Network Activation](https://mlsanity.com/paper/2507.11645/) (2025) - [The Geometric Structure of Models Learning Sparse Data](https://mlsanity.com/paper/2605.08464/) (2026) - [Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability](https://mlsanity.com/paper/2609.18078/) (2026) ## Abstract (arXiv) ''Grokking'' is a phenomenon where a neural network first memorizes training data and generalizes poorly, but then suddenly transitions to near-perfect generalization after prolonged training. While intriguing, this delayed generalization phenomenon compromises predictability and efficiency. Ideally, models should generalize directly without delay. To this end, this paper proposes GrokTransfer, a simple and principled method for accelerating grokking in training neural networks, based on the key observation that data embedding plays a crucial role in determining whether generalization is delayed. GrokTransfer first trains a smaller, weaker model to reach a nontrivial (but far from optimal) test performance. Then, the learned input embedding from this weaker model is extracted and used to initialize the embedding in the target, stronger model. We rigorously prove that, on a synthetic XOR task where delayed generalization always occurs in normal training, GrokTransfer enables the target model to generalize directly without delay. Moreover, we demonstrate that, across empirical studies of different tasks, GrokTransfer effectively reshapes the training dynamics and eliminates delayed generalization, for both fully-connected neural networks and Transformers.