mlsanityData to 7 Oct 2026

Small-scale proxies for large-scale Transformer training instabilities

arXiv 2309.14322 ↗First posted 25 Sep 2023cs.LGcited by 182

Not on the map. Why?

Abstract (arXiv)

Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the μParam (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

41 of 56 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016Layer Normalizationcited by 5,579
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
29 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  13. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  14. 2016Using the Output Embedding to Improve Language Modelscited by 266
  15. 2022Transformer Quality in Linear Timecited by 189
  16. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  17. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  18. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  19. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  20. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  21. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  22. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  23. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  24. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  25. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  26. 2020Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descentcited by 40
  27. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  28. 2023Effective Theory of Transformers at Initializationcited by 21
  29. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 60; 2025: 64; 2026: 55.

182 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2025Kimi K2: Open Agentic Intelligencecited by 523
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  8. 2024Massive Activations in Large Language Modelscited by 277
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  10. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  11. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  12. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
38 more
  1. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  3. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  4. 2024Scaling Laws for Precisioncited by 92
  5. 2024Stable LM 2 1.6B Technical Reportcited by 86
  6. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  7. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  8. 2024Hyper-Connectionscited by 81
  9. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  10. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  11. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  12. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  13. 2025Proteina: Scaling Flow-based Protein Structure Generative Modelscited by 63
  14. 2025Distillation Scaling Lawscited by 59
  15. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  16. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  17. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  18. 2025Unified Multimodal Discrete Diffusioncited by 46
  19. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  20. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  21. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 41
  22. 2024Scaling Laws For Diffusion Transformerscited by 41
  23. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  24. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  25. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  26. 2024Scaling Optimal LR Across Token Horizonscited by 35
  27. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  28. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
  29. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  30. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
  31. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  32. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  33. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  34. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  35. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  36. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  37. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  38. 2024Apple Intelligence Foundation Language Modelscited by 22

50 most cited of 182 All 182 (JSON)