mlsanityData to 7 Oct 2026

Emergence in non-neural models: grokking modular arithmetic via average gradient outer product

arXiv 2407.20199 ↗First posted 29 Jul 2024stat.ML cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Neural networks trained to solve modular arithmetic tasks exhibit grokking, a phenomenon where the test accuracy starts improving long after the model achieves 100% training accuracy in the training process. It is often taken as an example of "emergence", where model ability manifests sharply through a phase transition. In this work, we show that the phenomenon of grokking is not specific to neural networks nor to gradient descent-based optimization. Specifically, we show that this phenomenon occurs when learning modular arithmetic with Recursive Feature Machines (RFM), an iterative algorithm that uses the Average Gradient Outer Product (AGOP) to enable task-specific feature learning with general machine learning models. When used in conjunction with kernel machines, iterating RFM results in a fast transition from random, near zero, test accuracy to perfect test accuracy. This transition cannot be predicted from the training loss, which is identically zero, nor from the test loss, which remains constant in initial iterations. Instead, as we show, the transition is completely determined by feature learning: RFM gradually learns block-circulant features to solve modular arithmetic. Paralleling the results for RFM, we show that neural networks that solve modular arithmetic also learn block-circulant features. Furthermore, we present theoretical evidence that RFM uses such block-circulant features to implement the Fourier Multiplication Algorithm, which prior work posited as the generalizing solution neural networks learn on these tasks. Our results demonstrate that emergence can result purely from learning task-relevant features and is not specific to neural architectures nor gradient descent-based optimization methods. Furthermore, our work provides more evidence for AGOP as a key mechanism for feature learning in neural networks.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

36 of 46 references.

References · 36

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  4. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  5. 2017Implicit Regularization in Matrix Factorizationcited by 318
  6. 2021Deep learning: a statistical viewpointcited by 239
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  9. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  10. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  11. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  12. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
24 more
  1. 2019Robust Learning with Jacobian Regularizationcited by 127
  2. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  3. 2023Explaining grokking through circuit efficiencycited by 105
  4. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  5. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  6. 2023Grokking modular arithmeticcited by 77
  7. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  8. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  9. 2023Unifying Grokking and Double Descentcited by 49
  10. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  11. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  12. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  13. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  14. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  15. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  16. 2024Grokking Modular Polynomialscited by 18
  17. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16
  18. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  19. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  20. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  21. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  22. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  23. 2023Efficient Estimation of the Central Mean Subspace via Smoothed Gradient Outer Productscited by 7
  24. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 12.

26 papers.

Sort:

  1. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  2. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  3. 2025xRFM: Accurate, scalable, and interpretable feature learning models for tabular datacited by 14
  4. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  5. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  6. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  7. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  8. 2026The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behaviorcited by 3
  9. 2025A dynamic view of some anomalous phenomena in SGDcited by 2
  10. 2025Quiet Feature Learning in Algorithmic Taskscited by 2
  11. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  12. 2026Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmeticcited by 2
14 more
  1. 2026Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkincited by 2
  2. 2025Feature maps for the Laplacian kernel and its generalizationscited by 1
  3. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
  4. 2025On the Neural Feature Ansatz for Deep Neural Networkscited by 1
  5. 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
  6. 2026Grokking From Abstraction to Intelligencecited by 1
  7. 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
  8. 2026Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noisecited by 1
  9. 2026From Symmetry to Invariance: Learning Galois Equivalent Representations in Finite Fieldscited by 1
  10. 2025Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Additioncited by 0
  11. 2025Decomposing Prediction Mechanisms for In-Context Recallcited by 0
  12. 2026The Geometric Structure of Models Learning Sparse Datacited by 0
  13. 2026K-Inverse-RFM: A Modified RFM that Bridges the Gap to Neural Networks for Data-Corrupted Mathematical Taskscited by 0
  14. 2026A Spectral Theory of Grokking: Weight Decay induces Feature Learningcited by 0