mlsanityData to 7 Oct 2026

A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations

arXiv 2302.03025 ↗First posted 6 Feb 2023cs.LG cs.AI math.RTcited by 141

Not on the map. Why?

Abstract (arXiv)

Universality is a key hypothesis in mechanistic interpretability -- that different models learn similar features and circuits when trained on similar tasks. In this work, we study the universality hypothesis by examining how small neural networks learn to implement group composition. We present a novel algorithm by which neural networks may implement composition for any finite group via mathematical representation theory. We then show that networks consistently learn this algorithm by reverse engineering model logits and weights, and confirm our understanding using ablations. By studying networks of differing architectures trained on various groups, we find mixed evidence for universality: using our algorithm, we can completely characterize the family of circuits and features that networks learn on this task, but for a given network the precise circuits learned -- as well as the order they develop -- are arbitrary.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

22 of 45 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  11. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
10 more
  1. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  2. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  3. 2022Predictability and Surprise in Large Generative Modelscited by 209
  4. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  5. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  6. 2021Thinking Like Transformerscited by 182
  7. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  8. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  9. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  10. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 45; 2025: 31; 2026: 34.

141 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  6. 2023Localizing Model Behavior with Path Patchingcited by 182
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  8. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  9. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  10. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  11. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  12. 2023Explaining grokking through circuit efficiencycited by 105
38 more
  1. 2024Universal Neurons in GPT2 Language Modelscited by 100
  2. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  3. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  4. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  5. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  6. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  7. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  8. 2024Interpreting Attention Layer Outputs with Sparse Autoencoderscited by 43
  9. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  10. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 39
  11. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  12. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  13. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  14. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  15. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  16. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  17. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  18. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  19. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  20. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  21. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  22. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  23. 2023Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networkscited by 23
  24. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  25. 2023Grokking Group Multiplication with Cosetscited by 22
  26. 2025The Universal Weight Subspace Hypothesiscited by 22
  27. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  28. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  29. 2024Transformer Circuit Faithfulness Metrics are not Robustcited by 18
  30. 2023Can Transformers Learn to Solve Problems Recursively?cited by 17
  31. 2023Trainable Transformer in Transformercited by 17
  32. 2024Compact Proofs of Model Performance via Mechanistic Interpretabilitycited by 17
  33. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  34. 2024Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursioncited by 14
  35. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomialscited by 14
  36. 2024Learning biologically relevant features in a pathology foundation model using sparse autoencoderscited by 14
  37. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  38. 2025Implicit Reasoning in Transformers is Reasoning through Shortcutscited by 14

50 most cited of 141 All 141 (JSON)