mlsanityData to 7 Oct 2026

Formal Algorithms for Transformers

arXiv 2207.09238 ↗First posted 19 Jul 2022cs.LG cs.AI cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

18 of 34 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2016An overview of gradient descent optimization algorithmscited by 1,021
  11. 2015Automatic differentiation in machine learning: a surveycited by 846
  12. 2022A Generalist Agentcited by 802
6 more
  1. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  2. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  3. 2021A Survey of Transformerscited by 399
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  5. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  6. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 23; 2025: 14; 2026: 9.

70 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  3. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  4. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  5. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  6. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  7. 2024Evolving Code with A Large Language Modelcited by 42
  8. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  9. 2024Materials science in the era of large language models: a perspectivecited by 23
  10. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22
  11. 2023Can a Transformer Represent a Kalman Filter?cited by 20
  12. 2025Scaling Embedding Layers in Language Modelscited by 19
38 more
  1. 2023Can Transformers Learn to Solve Problems Recursively?cited by 17
  2. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  3. 2024Attention layers provably solve single-location regressioncited by 15
  4. 2023Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language modelcited by 14
  5. 2023RankFormer: Listwise Learning-to-Rank Using Listwide Labelscited by 9
  6. 2023Large Language Models Vote: Prompting for Rare Disease Identificationcited by 9
  7. 2024Wavelets Are All You Need for Autoregressive Image Generationcited by 9
  8. 2024Fundamental Limitations on Subquadratic Alternatives to Transformerscited by 9
  9. 2023Hierarchical Neural Simulation-Based Inference Over Event Ensemblescited by 8
  10. 2024Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architecturescited by 8
  11. 2023Explainability Techniques for Chemical Language Modelscited by 6
  12. 2024Variational Search Distributionscited by 6
  13. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  14. 2023Evaluating self-attention interpretability through human-grounded experimental protocolcited by 5
  15. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  16. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelismcited by 4
  17. 2024Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inferencecited by 4
  18. 2025T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoderscited by 4
  19. 2026Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioningcited by 4
  20. 2025Attention-based clusteringcited by 3
  21. 2026An Optimal Control Approach To Transformer Trainingcited by 3
  22. 2023Enhancing Object Detection in Ancient Documents with Synthetic Data Generation and Transformer-Based Modelscited by 2
  23. 2023Natural Language Processing for Financial Regulationcited by 2
  24. 2024Learning Randomized Algorithms with Transformerscited by 2
  25. 2024Towards a Categorical Foundation of Deep Learning: A Surveycited by 2
  26. 2025Modes of Sequence Models and Learning Coefficientscited by 2
  27. 2025Generative Bayesian Optimization: Generative Models as Acquisition Functionscited by 2
  28. 2022Perceived personality state estimation in dyadic and small group interaction with deep learning methodscited by 1
  29. 2024Increasing transformer token length with a Maximum Entropy Principle Methodcited by 1
  30. 2024Broadcast Product: Redefining Shape-aligned Element-wise Multiplication and Beyondcited by 1
  31. 2024Notes on the Mathematical Structure of GPT LLM Architecturescited by 1
  32. 2025Emergence of Computational Structure in a Neural Network Physics Simulatorcited by 1
  33. 2025Amortized Active Generation of Pareto Setscited by 1
  34. 2026What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformerscited by 1
  35. 2026Attention-based graph neural networks: a surveycited by 1
  36. 2022GAMMT: Generative Ambiguity Modeling Using Multiple Transformerscited by 0
  37. 2023Coinductive guide to inductive transformer headscited by 0
  38. 2023Transformadores: Fundamentos teoricos y Aplicacionescited by 0

50 most cited of 70 All 70 (JSON)