# On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking

arXiv: https://arxiv.org/abs/2602.16849 · HTML: https://mlsanity.com/paper/2602.16849/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2026.

## Built on by (most cited first, 7 of 7)

- [Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization](https://mlsanity.com/paper/2605.05683/) (2026)
- [Neural Networks Provably Learn Spectral Representations for Group Composition](https://mlsanity.com/paper/2606.02993/) (2026)
- [Assign and Add: A Mechanistic Study of Compositional Arithmetic](https://mlsanity.com/paper/2605.31497/) (2026)
- [A Counterexample to Fourier Alignment in Single-Neuron Modular Addition](https://mlsanity.com/paper/2608.04451/) (2026)
- [Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs](https://mlsanity.com/paper/2608.24007/) (2026)
- [Symmetry without a manifold: intrinsic dimension on orbits](https://mlsanity.com/paper/2609.17926/) (2026)
- [A Spectral Theory of Grokking: Weight Decay induces Feature Learning](https://mlsanity.com/paper/2609.26679/) (2026)

## Abstract (arXiv)

We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.