# Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers

arXiv: https://arxiv.org/abs/2608.07436 · HTML: https://mlsanity.com/paper/2608.07436/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)
- [Muon Outperforms Adam in Tail-End Associative Memory Learning](https://mlsanity.com/paper/2509.26030/) (2025)
- [Muon Optimizer Accelerates Grokking](https://mlsanity.com/paper/2504.16041/) (2025)
- [Grokking and Generalization Collapse: Insights from textttHTSR theory](https://mlsanity.com/paper/2506.04434/) (2025)
- [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](https://mlsanity.com/paper/2605.06152/) (2026)
- [The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior](https://mlsanity.com/paper/2604.13082/) (2026)
- [Muon Learns More Robust and Transferable Features than Adam](https://mlsanity.com/paper/2606.09658/) (2026)
- [Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent](https://mlsanity.com/paper/2605.27078/) (2026)
- [Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment](https://mlsanity.com/paper/2310.05644/) (2023)
- [The Active Ingredient in Muon's Grokking](https://mlsanity.com/paper/2607.20512/) (2026)

## Built on by (most cited first, 2 of 2)

- [Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability](https://mlsanity.com/paper/2609.18078/) (2026)
- [Faynt: Scaling and Optimizing Policies for Competitive Melee](https://mlsanity.com/paper/2610.02144/) (2026)

## Abstract (arXiv)

Muon-trained modular-arithmetic transformers can lose accuracy while retaining linearly decodable task information. Adjacent swaps localize five captured unnormalized failures to AdamW readout updates. Multiplying the actual readout displacement by the large feature mean produces a class-dependent logit offset shared across inputs that nearly reproduces each failure. Training-only decoders recover 98.20-100% held-out accuracy. Correcting cross-entropy derivative errors stabilizes five matched branches through step 100,000; four prospective accurate-CE RMS runs fail through embedding updates.