mlsanityData to 14 Oct 2026

Where Grokking Happens: Distributed Utility and Fourier Recoding Without a Module Switch

arXiv 2609.17571 ↗First posted 29 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Where in a Transformer is the change from memorization to generalization functionally expressed? We introduce Transition Games--behavior-aligned exact activation games with paired non-generalizing controls--and find distributed utility gain with a prospective block-0 attention bias; selected degree-two modes account for 67--92% of its addition contrast across replacement games, and a disjoint exact path study confirms that block-1 MLP mediates more of their effect than all other tested downstream paths in 12/12 pairs. The sharper "MLP memorizes, attention generalizes" prediction instead reverses (-.331 bits/example at the memory anchor; 0/12 in the predicted direction), while routing onset, global rank collapse, and a prime-invariant architecture ridge also fail, identifying grokking here as spectral recoding of an existing distributed circuit rather than a module switch.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

23 of 31 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  8. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  9. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  10. 2023Explaining grokking through circuit efficiencycited by 111
  11. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  12. 2023Can Neural Network Memorization Be Localized?cited by 69
11 more
  1. 2025How much do language models memorize?cited by 61
  2. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 36
  3. 2024Localizing Paragraph Memorization in Language Modelscited by 31
  4. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 21
  5. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  6. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  7. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  8. 2026Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speedscited by 4
  9. 2025ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behaviorcited by 3
  10. 2026Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformerscited by 1
  11. 2026Grokking as Structural Inference: Transformers Need Bayesian Lottery Ticketscited by 1

Built on by

Not cited yet.