Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Not on the map. Why?
Abstract (arXiv)
Muon-trained modular-arithmetic transformers can lose accuracy while retaining linearly decodable task information. Adjacent swaps localize five captured unnormalized failures to AdamW readout updates. Multiplying the actual readout displacement by the large feature mean produces a class-dependent logit offset shared across inputs that nearly reproduces each failure. Training-only decoders recover 98.20-100% held-out accuracy. Correcting cross-entropy derivative errors stabilizes five matched branches through step 100,000; four prospective accurate-CE RMS runs fail through embedding updates.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
18 of 27 references.
References · 18
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2025Muon is Scalable for LLM Trainingcited by 381
- 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
- 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
- 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
- 2023Explaining grokking through circuit efficiencycited by 105
- 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
- 2025Grokking at the Edge of Numerical Stabilitycited by 41
- 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
- 2025Muon Optimizer Accelerates Grokkingcited by 11
- 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
6 more
- 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
- 2026The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behaviorcited by 3
- 2026Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descentcited by 2
- 2023Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignmentcited by 1
- 2026Muon Learns More Robust and Transferable Features than Adamcited by 1
- 2026The Active Ingredient in Muon's Grokkingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.