mlsanityData to 14 Oct 2026

Progress Measures for Grokking on Real-world Tasks

arXiv 2405.12755 ↗First posted 21 May 2024cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Grokking, a phenomenon where machine learning models generalize long after overfitting, has been primarily observed and studied in algorithmic tasks. This paper explores grokking in real-world datasets using deep neural networks for classification under the cross-entropy loss. We challenge the prevalent hypothesis that the L2 norm of weights is the primary cause of grokking by demonstrating that grokking can occur outside the expected range of weight norms. To better understand grokking, we introduce three new progress measures: activation sparsity, absolute weight entropy, and approximate local circuit complexity. These measures are conceptually related to generalization and demonstrate a stronger correlation with grokking in real-world datasets compared to weight norms. Our findings suggest that while weight norms might usually correlate with grokking and our progress measures, they are not causative, and our proposed measures provide a better understanding of the dynamics of grokking.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

18 of 24 references.

References · 18

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  3. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  4. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  5. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  6. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  7. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  8. 2023Explaining grokking through circuit efficiencycited by 111
  9. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  10. 2017On weight initialization in deep neural networkscited by 67
  11. 2018Entropy and mutual information in models of deep neural networkscited by 51
  12. 2024Deep Networks Always Grok and Here is Whycited by 51
6 more
  1. 2018Entropy-Constrained Training of Deep Neural Networkscited by 8
  2. 2022Entropy-based Stability-Plasticity for Lifelong Learningcited by 8
  3. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  4. 2021The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networkscited by 3
  5. 2024The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depthcited by 3
  6. 2020Complexity for deep neural networks and other characteristics of deep feature representationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
  2. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  3. 2026The Weight Norm Sets the Grokking Timescale: A Causal Delay Lawcited by 4
  4. 2025NeuralGrok: Accelerate Grokking by Neural Gradient Transformationcited by 2
  5. 2026Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descentcited by 2
  6. 2026Grokking as Structural Inference: Transformers Need Bayesian Lottery Ticketscited by 1
  7. 2025Tracing the Path to Grokking: Embeddings, Dropout, and Network Activationcited by 0
  8. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0