mlsanityData to 7 Oct 2026

Effective Theory of Transformers at Initialization

arXiv 2304.02034 ↗First posted 4 Apr 2023cs.LG cs.CL hep-thcited by 21

Not on the map. Why?

Abstract (arXiv)

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 46 of its 56 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

46 of 56 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2016Layer Normalizationcited by 5,579
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
34 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2017Mixed Precision Trainingcited by 1,135
  14. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  15. 2021Scaling Vision Transformerscited by 993
  16. 2016Deep Networks with Stochastic Depthcited by 973
  17. 2017Deep Neural Networks as Gaussian Processescited by 617
  18. 2020Recipes for building an open-domain chatbotcited by 593
  19. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  20. 2021Early Convolutions Help Transformers See Bettercited by 395
  21. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  22. 2016On the Expressive Power of Deep Neural Networkscited by 302
  23. 2020The Pushshift Reddit Datasetcited by 289
  24. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  25. 2016Using the Output Embedding to Improve Language Modelscited by 266
  26. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  27. 2018A Simple Method for Commonsense Reasoningcited by 223
  28. 2016Deep Information Propagationcited by 194
  29. 2021The Principles of Deep Learning Theorycited by 169
  30. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  31. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  32. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  33. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  34. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 4; 2025: 6; 2026: 2.

21 papers.

Sort:

By area:

  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  3. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  4. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  5. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  6. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  7. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
  8. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  9. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  10. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 21
  11. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 15
  12. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
9 more
  1. 2023Structures of Neural Network Effective Theoriescited by 13
  2. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  3. 2023Feature Learning and Generalization in Deep Networks with Orthogonal Weightscited by 5
  4. 2025Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscited by 5
  5. 2025Fermions and Supersymmetry in Neural Network Field Theoriescited by 5
  6. 2024Principled Architecture-aware Scaling of Hyperparameterscited by 4
  7. 2023Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activationscited by 3
  8. 2025Stochastic Approximation with Block Coordinate Optimal Stepsizescited by 3
  9. 2026Gaussian Equivalence for Multi-Head Self-Attentioncited by 0