mlsanityData to 14 Oct 2026

Learning Modular Exponentiation with Transformers

arXiv 2506.23679 ↗First posted 30 Jun 2025cs.LG cs.AI cs.CRcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Modular exponentiation is crucial to number theory and cryptography, yet remains largely unexplored from a mechanistic interpretability standpoint. We train a 4-layer encoder-decoder Transformer model to perform this operation and investigate the emergence of numerical reasoning during training. Utilizing principled sampling strategies, PCA-based embedding analysis, and activation patching, we examine how number-theoretic properties are encoded within the model. We find that reciprocal operand training leads to strong performance gains, with sudden generalization across related moduli. These synchronized accuracy surges reflect grokking-like dynamics, suggesting the model internalizes shared arithmetic structure. We also find a subgraph consisting entirely of attention heads in the final layer sufficient to achieve full performance on the task of regular exponentiation. These results suggest that transformer models learn modular arithmetic through specialized computational circuits, paving the way for more interpretable and efficient neural approaches to modular exponentiation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 12 of its 23 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

12 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models2 Jul
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  3. 2023Progress measures for grokking via mechanistic interpretability12 Jan
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 8

  1. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  2. 2024How to use and interpret activation patchingcited by 214
  3. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  4. 2023Understanding Addition in Transformerscited by 34
  5. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  6. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  7. 2024Grokking Modular Polynomialscited by 18
  8. 2022Murmurations of elliptic curvescited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
  2. 2024Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials
  3. 2024Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
  4. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks
  5. 2026Transformers Learn the Mestre-Nagao Heuristic
  6. 2025Modular Arithmetic: Language Models Solve Math Digit by Digit
  7. 2022Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation
  8. 2024Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures