mlsanityData to 14 Oct 2026

A rationale from frequency perspective for grokking in training neural network

arXiv 2405.17479 ↗First posted 24 May 2024cs.LG cs.NE stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Grokking is the phenomenon where neural networks NNs initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a frequency perspective to explain the emergence of this phenomenon in NNs. The core insight is that the networks initially learn the less salient frequency components present in the test data. We observe this phenomenon across both synthetic and real datasets, offering a novel viewpoint for elucidating the grokking phenomenon by characterizing it through the lens of frequency dynamics during the training process. Our empirical frequency-based analysis sheds new light on understanding the grokking phenomenon and its underlying mechanisms.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 32 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

30 of 32 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  7. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  8. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  9. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  11. 2018Training behavior of deep neural network in frequency domaincited by 174
  12. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
18 more
  1. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  2. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  3. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  4. 2023Explaining grokking through circuit efficiencycited by 111
  5. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  6. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 94
  7. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  8. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  9. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  10. 2022Overview frequency principle/spectral bias in deep learningcited by 69
  11. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  12. 2019Machine Learning from a Continuous Viewpointcited by 50
  13. 2019Theory of the Frequency Principle for General Deep Neural Networkscited by 46
  14. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  15. 2019Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networkscited by 20
  16. 2020On the exact computation of linear frequency principle dynamics and its generalizationcited by 16
  17. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14
  18. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  2. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
  3. 2026On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regimecited by 1