A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
Not on the map. Why?
Abstract (arXiv)
We give a negative solution to MAIS-O60. We first construct an example in which an initially active ReLU neuron becomes completely inactive in finite time and thereafter remains frozen at a limit whose Fourier energy is equally distributed among all nonzero real frequency classes. The counterexample holds on an open set of initial conditions and therefore occurs with positive probability under Gaussian initialization. An appendix prepared by GPT-5.6 Sol strengthens the counterexample by showing that the same failure can occur for every Clarke trajectory from an open set of initial conditions, under the convention ReLU'(0)=0, for smooth dead-zone approximations of ReLU, and for fixed-step full-batch gradient descent. Thus, single-frequency alignment is not a general consequence of training a single neuron on modular addition.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 8 of its 15 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
8 of 15 references.
References · 8
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 831
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 652
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 312
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 175
- 2020Directional convergence and alignment in deep learningcited by 174
- 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
- 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
- 2026On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokkingcited by 7
Built on by
Not cited yet.