mlsanityData to 14 Oct 2026

Grokking Group Multiplication with Cosets

arXiv 2312.06581 ↗First posted 11 Dec 2023cs.LG cs.AI math.RTcited by 24

Not on the map. Why?

Abstract (arXiv)

The complex and unpredictable nature of deep neural networks prevents their safe use in many high-stakes applications. There have been many techniques developed to interpret deep neural networks, but all have substantial limitations. Algorithmic tasks have proven to be a fruitful test ground for interpreting a neural network end-to-end. Building on previous work, we completely reverse engineer fully connected one-hidden layer networks that have ``grokked'' the arithmetic of the permutation groups S5 and S6. The models discover the true subgroup structure of the full group and converge on neural circuits that decompose the group arithmetic using the permutation group's subgroups. We relate how we reverse engineered the model's mechanisms and confirmed our theory was a faithful description of the circuit's functionality. We also draw attention to current challenges in conducting interpretability research by comparing our work to Chughtai et al. [4] which alleges to find a different algorithm for this same problem.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 35 of its 74 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

35 of 74 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  7. 2018Sanity Checks for Saliency Mapscited by 1,156
  8. 2019Attention is not Explanationcited by 929
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  10. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
23 more
  1. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  2. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  3. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  4. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
  5. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  6. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  7. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  8. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  9. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  10. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  11. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  12. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  13. 2023Explaining grokking through circuit efficiencycited by 111
  14. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  15. 2021An Interpretability Illusion for BERTcited by 97
  16. 2023Grokking modular arithmeticcited by 83
  17. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  18. 2023Discovering Variable Binding Circuitry with Desideratacited by 34
  19. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  20. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  21. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  22. 2023Can Transformers Learn to Solve Problems Recursively?cited by 17
  23. 2023Neural Discovery of Permutation Subgroupscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 15.

24 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  3. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  4. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  5. 2024Towards a unified and verified understanding of group-operation networkscited by 10
  6. 2026Sequential Group Composition: A Window into the Mechanics of Deep Learningcited by 10
  7. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  8. 2025In-Context Algebracited by 5
  9. 2025On the geometry and topology of representations: the manifolds of modular additioncited by 3
  10. 2026Can Neural Networks Learn Small Algebraic Worlds? An Investigation Into the Group-theoretic Structures Learned By Narrow Models Trained To Predict Group Operationscited by 3
  11. 2024Machines and Mathematical Mutations: Using GNNs to Characterize Quiver Mutation Classescited by 2
  12. 2025Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.iicited by 2
12 more
  1. 2026Grokking Finite-Dimensional Algebracited by 2
  2. 2026Spectral Edge Dynamics Reveal Functional Modes of Learningcited by 2
  3. 2026Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descentcited by 2
  4. 2026Learned Structure Shows Up Earlier in the Task's Own Basis: A Controlled Look at a Synchronisation Precursor in Grokkingcited by 2
  5. 2026Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuitscited by 2
  6. 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
  7. 2026From Symmetry to Invariance: Learning Galois Equivalent Representations in Finite Fieldscited by 1
  8. 2026The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Compositioncited by 1
  9. 2026PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatoricscited by 1
  10. 2026The Discrete-Log Clock: How a Transformer Learns Modular Multiplicationcited by 0
  11. 2026Symmetry without a manifold: intrinsic dimension on orbitscited by 0
  12. 2026Tracking States or Tracking Cosets? An Algebraic Account of Learned State Trackingcited by 0