mlsanityData to 14 Oct 2026

Linear algebra with transformers

arXiv 2112.01898 ↗First posted 3 Dec 2021cs.LG cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

Transformers can learn to perform numerical computations from examples only. I study nine problems of linear algebra, from basic matrix operations to eigenvalue decomposition and inversion, and introduce and discuss four encoding schemes to represent real numbers. On all problems, transformers trained on sets of random matrices achieve high accuracies (over 90%). The models are robust to noise, and can generalize out of their training distribution. In particular, models trained to predict Laplace-distributed eigenvalues generalize to different classes of matrices: Wigner matrices or matrices with positive eigenvalues. The reverse is not true.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 37 of its 68 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

37 of 68 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2015Pointer Networkscited by 895
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
25 more
  1. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  2. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  3. 2019Deep Learning for Symbolic Mathematicscited by 270
  4. 2021Combinatorial optimization and reasoning with graph neural networkscited by 268
  5. 2021Neural Symbolic Regression that Scalescited by 192
  6. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  7. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 132
  8. 2020Fast Transformers with Clustered Attentioncited by 123
  9. 2015Neural GPUs Learn Algorithmscited by 121
  10. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 88
  11. 2017Failures of Gradient-Based Deep Learningcited by 80
  12. 2018Neural Arithmetic Logic Unitscited by 70
  13. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  14. 2020Learning continuous-time PDEs from sparse data with graph neural networkscited by 53
  15. 2020Teaching Temporal Logics to Neural Networkscited by 51
  16. 2021Multiplying Matrices Without Multiplyingcited by 30
  17. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  18. 2015Grid Long Short-Term Memorycited by 20
  19. 2015Learning Simple Algorithms from Examplescited by 12
  20. 2019The Use of Deep Learning for Symbolic Integration: A Review of (Lample and Charton, 2019)cited by 11
  21. 2021Transformer-based Machine Learning for Fast SAT Solvers and Logic Synthesiscited by 9
  22. 2019Solving Math Word Problems with Double-Decoder Transformercited by 7
  23. 2021Solving Arithmetic Word Problems with Transformers and Preprocessing of Problem Textcited by 7
  24. 2015Moment-Based Spectral Analysis of Random Graphs with Given Expected Degreescited by 1
  25. 2019Learning Numeracy: Binary Arithmetic with Neural Turing Machinescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 15; 2024: 17; 2025: 18; 2026: 4.

63 papers.

Sort:

By area:

  1. 2023Looped Transformers as Programmable Computerscited by 237
  2. 2022End-to-end symbolic regression with transformerscited by 203
  3. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  4. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  5. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  6. 2023Teaching Arithmetic to Small Transformerscited by 123
  7. 2022Tutorial on amortized optimizationcited by 83
  8. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  9. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  10. 2023Length Generalization in Arithmetic Transformerscited by 66
  11. 2023ODEFormer: Symbolic Regression of Dynamical Systems with Transformerscited by 62
  12. 2023Positional Description Matters for Transformers Arithmeticcited by 56
38 more
  1. 2022Deep Symbolic Regression for Recurrent Sequencescited by 54
  2. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 47
  3. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 46
  4. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  5. 2024OmniPred: Language Models as Universal Regressorscited by 34
  6. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  7. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  8. 2024Large Language Model-Based Evolutionary Optimizer: Reasoning with elitismcited by 27
  9. 2022What is my math transformer doing? -- Three results on interpretability and generalizationcited by 19
  10. 2024Transforming the Bootstrap: Using Transformers to Compute Scattering Amplitudes in Planar N = 4 Super Yang-Mills Theorycited by 18
  11. 2023PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformerscited by 16
  12. 2023Is deep learning a useful tool for the pure mathematician?cited by 14
  13. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 14
  14. 2025Performance Prediction for Large Systems via Text-to-Text Regressioncited by 14
  15. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  16. 2024Meta-Designing Quantum Experiments with Language Modelscited by 12
  17. 2025Decoding-based Regressioncited by 12
  18. 2023SALSA PICANTE: a machine learning attack on LWE with binary secretscited by 11
  19. 2024Emergent properties with repeated examplescited by 11
  20. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  21. 2025A Neural Symbolic Model for Space Physicscited by 11
  22. 2023Learning to Compute Gr"obner Basescited by 7
  23. 2024Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errorscited by 7
  24. 2024Models That Prove Their Own Correctnesscited by 7
  25. 2025Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithmscited by 7
  26. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  27. 2024Machine-learning emergent spacetime from linear response in future tabletop quantum gravity experimentscited by 5
  28. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  29. 2025Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Taskcited by 4
  30. 2023Tracing and Manipulating Intermediate Values in Neural Math Problem Solverscited by 3
  31. 2025Learning the symmetric group: large from smallcited by 3
  32. 2025From Black Box to Bijection: Interpreting Machine Learning to Build a Zeta Map Algorithmcited by 3
  33. 2022TgDLF2.0: Theory-guided deep-learning for electrical load forecasting via Transformer and transfer learningcited by 2
  34. 2024Machine learning for modular multiplicationcited by 2
  35. 2024Graph Transformers Dream of Electric Flowcited by 2
  36. 2021Deep Learning and Mathematical Intuition: A Review of (Davies et al. 2021)cited by 1
  37. 2024Multi-objective Representation for Numbers in Clinical Narratives: A CamemBERT-Bio-Based Alternative to Large-Scale LLMscited by 1
  38. 2025Generating particle physics Lagrangians with transformerscited by 1

50 most cited of 63 All 63 (JSON)