mlsanityData to 7 Oct 2026

Progress measures for grokking via mechanistic interpretability

arXiv 2301.05217 ↗First posted 12 Jan 2023cs.LG cs.AIcited by 822

Not on the map. Why?

Cited by 56 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous textitprogress measures that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of ``grokking'' exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

Key facts. Cited by 822 papers in the map (data through 7 Oct 2026). Builds on 14 of its 27 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

14 of 27 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  11. 2022Predictability and Surprise in Large Generative Modelscited by 209
  12. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
2 more
  1. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  2. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 109; 2024: 172; 2025: 236; 2026: 305.

822 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  3. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  8. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  9. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  11. 2023Large Language Model Alignment: A Surveycited by 241
  12. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
38 more
  1. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  2. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  3. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  4. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  5. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  6. 2023Model evaluation for extreme riskscited by 167
  7. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  9. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  10. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  11. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  12. 2023Getting aligned on representational alignmentcited by 142
  13. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  14. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  15. 2023The Quantization Model of Neural Scalingcited by 137
  16. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  17. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  18. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  19. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  20. 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
  21. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  22. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  23. 2023Explaining grokking through circuit efficiencycited by 105
  24. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  25. 2025Agentic Large Language Models, a surveycited by 105
  26. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  27. 2023How do Language Models Bind Entities in Context?cited by 102
  28. 2024Universal Neurons in GPT2 Language Modelscited by 100
  29. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  30. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  31. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  32. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  33. 2025Model Organisms for Emergent Misalignmentcited by 84
  34. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  35. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  36. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  37. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  38. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78

50 most cited of 822 All 822 (JSON)