mlsanityData to 7 Oct 2026

Grokking Modular Polynomials

arXiv 2406.03495 ↗First posted 5 Jun 2024cs.LG cond-mat.dis-nn hep-thcited by 18

Not on the map. Why?

Abstract (arXiv)

Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and training strategies. On the other hand, an analytical solution for the weights of Multi-layer Perceptron (MLP) networks that generalize on the modular addition task is known in the literature. In this work, we (i) extend the class of analytical solutions to include modular multiplication as well as modular addition with many terms. Additionally, we show that real networks trained on these datasets learn similar solutions upon generalization (grokking). (ii) We combine these "expert" solutions to construct networks that generalize on arbitrary modular polynomials. (iii) We hypothesize a classification of modular polynomials into learnable and non-learnable via neural networks training; and provide experimental evidence supporting our claims.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

15 of 21 references.

References · 15

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  8. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  9. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  10. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  11. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  12. 2023Unifying Grokking and Double Descentcited by 49
3 more
  1. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  2. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  3. 2020High performance SIMD modular arithmetic for polynomial evaluationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 9.

18 papers.

Sort:

  1. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  2. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  3. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  4. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  5. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  6. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  7. 2025Learning words in groups: fusion algebras, tensor ranks and grokkingcited by 3
  8. 2026Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activationscited by 2
  9. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 1
  10. 2026Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutionscited by 1
  11. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  12. 2025Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Additioncited by 0
6 more
  1. 2025Learning Modular Exponentiation with Transformerscited by 0
  2. 2026Learning Large-Scale Modular Addition with an Auxiliary Moduluscited by 0
  3. 2026The Discrete-Log Clock: How a Transformer Learns Modular Multiplicationcited by 0
  4. 2026Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performancecited by 0
  5. 2026Structured Features Overfit Where Random Features Grokcited by 0
  6. 2026Symmetry without a manifold: intrinsic dimension on orbitscited by 0