# Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

arXiv: https://arxiv.org/abs/2605.08119 · HTML: https://mlsanity.com/paper/2605.08119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 7 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Builds on (most cited first, 6 of 6)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking](https://mlsanity.com/paper/2509.21519/) (2025)
- [Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking](https://mlsanity.com/paper/2602.16746/) (2026)

## Abstract (arXiv)

Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix  B = (widetildeF^top widetildeF + η I)-1  during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries  Bjℓ , producing an effective repulsive force that drives them apart. However, the theorem does not specify when this mechanism becomes empirically observable, nor whether it leaves a measurable spectral signature in the parameter updates. We test this directly on Tian's modular addition setup ( M = 71 ,  K = 2048 , MSE loss) and observe a clear structure-mechanism dissociation. The predicted sign rule holds robustly on the top-200 most-similar feature pairs across activations (empirical sign-match rising from 0.865 to 0.985 on  σ = x2  across 5 seeds, and saturating at 1.000 on  σ = operatornameReLU ). However, the spectral signature in the parameter updates is strongly activation-dependent. With  σ = x2 , a simple slope detector on the rolling eigengap  σ2 / σ3  of  Delta W  fires in 15/15 grokking seeds at epoch 174 (IQR [173,174]) and in 0/15 non-grokking controls, with 229 ×  late-stage magnitude separation; the spectrum is rank-2. In contrast, with  σ = operatornameReLU , the detector never fires and the spectrum remains effectively rank-1. This dissociation aligns with Tian's Theorem 5 distinction between focused (power-law) and spreading (ReLU) memorization: while the sign structure of  B  depends only on  widetildeF^top widetildeF , how feature repulsion translates into weight updates critically depends on the activation derivative  σ' .