mlsanityData to 14 Oct 2026

What is my math transformer doing? -- Three results on interpretability and generalization

arXiv 2211.00170 ↗First posted 31 Oct 2022cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

This paper investigates the failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition. I show that incorrect model predictions still retain deep mathematical properties of the solution (e.g. correct eigenvalues, unit norm of eigenvectors), and that almost all model failures can be attributed to, and predicted from, properties of the problem or solution. This demonstrates that, when in doubt, math transformers do not hallucinate absurd solutions (as was sometimes proposed) but remain ``roughly right''. I also show that the careful choice of a training dataset can accelerate training, while allowing the model to generalize out of its training distribution, invalidating the idea that transformers ``merely interpolate'' from memorized examples.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

11 of 14 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  5. 2019Deep Learning for Symbolic Mathematicscited by 270
  6. 2021Neural Symbolic Regression that Scalescited by 192
  7. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  8. 2021Linear algebra with transformerscited by 63
  9. 2020It's Not What Machines Can Learn, It's What We Cannot Teachcited by 29
  10. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  11. 2022Simplifying Polylogarithms with Machine Learningcited by 15

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 8; 2025: 4.

19 papers.

Sort:

By area:

  1. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  3. 2023Teaching Arithmetic to Small Transformerscited by 123
  4. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  5. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  6. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 40
  7. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  8. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  9. 2023Controllable Neural Symbolic Regressioncited by 20
  10. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  11. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  12. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
7 more
  1. 2024Theoretical Constraints on the Expressive Power of mathsfRoPE-based Tensor Attention Transformerscited by 12
  2. 2024Modern Hopfield Networks Require Chain-of-Thought to Solve mathsfNC1-Hard Problemscited by 9
  3. 2025Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descentcited by 8
  4. 2023Learning to Compute Gr"obner Basescited by 7
  5. 2025Learning the symmetric group: large from smallcited by 3
  6. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  7. 2025Generating particle physics Lagrangians with transformerscited by 1