mlsanityData to 14 Oct 2026

The Transformer Cookbook

arXiv 2510.00368 ↗First posted 1 Oct 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We present the transformer cookbook: a collection of techniques for directly encoding algorithms into a transformer's parameters. This work addresses the steep learning curve of such endeavors, a problem exacerbated by a fragmented literature where key results are scattered across numerous papers. In particular, we synthesize this disparate body of findings into a curated set of recipes that demonstrate how to implement everything from basic arithmetic in feed-forward layers to complex data routing via self-attention. Our mise en place of formulations is for both newcomers seeking an accessible entry point and experts in need of a systematic reference. This unified presentation of transformer constructions provides a foundation for future work spanning theoretical research in computational complexity to empirical investigations in architecture design and interpretability.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

38 of 52 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Layer Normalizationcited by 5,669
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2020On Layer Normalization in the Transformer Architecturecited by 922
  10. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  12. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
26 more
  1. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  2. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  3. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  4. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  5. 2021Thinking Like Transformerscited by 184
  6. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  7. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  8. 2023Representational Strengths and Limitations of Transformerscited by 133
  9. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  10. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  11. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  12. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  13. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
  14. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  15. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
  16. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
  17. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 53
  18. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 49
  19. 2020Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descentcited by 40
  20. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  21. 2023Logical Languages Accepted by Transformer Encoders with Hard Attentioncited by 38
  22. 2023On the Expressivity Role of LayerNorm in Transformers' Attentioncited by 33
  23. 2024One-layer transformers fail to solve the induction heads taskcited by 32
  24. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  25. 2024Simulating Hard Attention Using Soft Attentioncited by 15
  26. 2024Transformers as Transducerscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Why Are Linear RNNs More Parallelizable?cited by 8
  2. 2026Parity, Sensitivity, and Transformerscited by 5
  3. 2026Length Generalization Bounds for Transformerscited by 4
  4. 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
  5. 2026Transformer Approximations from ReLUscited by 0
  6. 2026On the Expressive Power of Transformerscited by 0