mlsanityData to 7 Oct 2026

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

arXiv 2306.13253 ↗First posted 23 Jun 2023cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

This paper focuses on predicting the occurrence of grokking in neural networks, a phenomenon in which perfect generalization emerges long after signs of overfitting or memorization are observed. It has been reported that grokking can only be observed with certain hyper-parameters. This makes it critical to identify the parameters that lead to grokking. However, since grokking occurs after a large number of epochs, searching for the hyper-parameters that lead to it is time-consuming. In this paper, we propose a low-cost method to predict grokking without training for a large number of epochs. In essence, by studying the learning curve of the first few epochs, we show that one can predict whether grokking will occur later on. Specifically, if certain oscillations occur in the early epochs, one can expect grokking to occur if the model is trained for a much longer period of time. We propose using the spectral signature of a learning curve derived by applying the Fourier transform to quantify the amplitude of low-frequency components to detect the presence of such oscillations. We also present additional experiments aimed at explaining the cause of these oscillations and characterizing the loss landscape.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

37 of 64 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  10. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2014Convex Optimization: Algorithms and Complexitycited by 569
25 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2017Three Factors Influencing Minima in SGDcited by 270
  10. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  11. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  12. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  13. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  14. 2014Qualitatively characterizing neural network optimization problemscited by 186
  15. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  16. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  17. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  18. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  19. 2021The Modern Mathematics of Deep Learningcited by 63
  20. 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
  21. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  22. 2016Gradient Descent Converges to Minimizerscited by 25
  23. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  24. 2016An empirical analysis of the optimization of deep network loss surfacescited by 21
  25. 2017Exploring loss function topology with cyclical learning ratescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 11; 2025: 5; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  2. 2023Explaining grokking through circuit efficiencycited by 105
  3. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  4. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  5. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  6. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  7. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  8. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  9. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  10. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  11. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 22
  12. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
19 more
  1. 2024Grokking Modular Polynomialscited by 18
  2. 2024Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transitioncited by 16
  3. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  4. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  5. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  6. 2024Grokking at the Edge of Linear Separabilitycited by 11
  7. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  8. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Testcited by 10
  9. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  10. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  11. 2026Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimizationcited by 3
  12. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 1
  13. 2025When Data Falls Short: Grokking Below the Critical Thresholdcited by 1
  14. 2026Grokking Finite-Dimensional Algebracited by 1
  15. 2025Tracing the Path to Grokking: Embeddings, Dropout, and Network Activationcited by 0
  16. 2025Grokked Models are Better Unlearnerscited by 0
  17. 2026Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governancecited by 0
  18. 2026Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performancecited by 0
  19. 2026Capability Emergence Can Be Forecast: Per-Seed, In Advance, With Calibrated Intervals, Certified False Alarms, and a Blind Pre-Registered Gatecited by 0