mlsanityData to 7 Oct 2026

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

arXiv 2305.14342 ↗First posted 23 May 2023cs.LG cs.CL math.OCcited by 260

Not on the map. Why?

Abstract (arXiv)

Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training. Adam and its variants have been state-of-the-art for years, and more sophisticated second-order (Hessian-based) optimizers often incur too much per-step overhead. In this paper, we propose Sophia, Second-order Clipped Stochastic Optimization, a simple scalable second-order optimizer that uses a light-weight estimate of the diagonal Hessian as the pre-conditioner. The update is the moving average of the gradients divided by the moving average of the estimated Hessian, followed by element-wise clipping. The clipping controls the worst-case update size and tames the negative impact of non-convexity and rapid change of Hessian along the trajectory. Sophia only estimates the diagonal Hessian every handful of iterations, which has negligible average per-step time and memory overhead. On language modeling with GPT models of sizes ranging from 125M to 1.5B, Sophia achieves a 2x speed-up compared to Adam in the number of steps, total compute, and wall-clock time, achieving the same perplexity with 50% fewer steps, less total compute, and reduced wall-clock time. Theoretically, we show that Sophia, in a much simplified setting, adapts to the heterogeneous curvatures in different parameter dimensions, and thus has a run-time bound that does not depend on the condition number of the loss.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). Builds on 47 of its 83 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

47 of 83 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
35 more
  1. 2021Convex optimizationcited by 2,479
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  13. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  14. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  15. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  16. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  17. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  18. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  19. 2020Understanding the Difficulty of Training Transformerscited by 210
  20. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  21. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  22. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  23. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  24. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  25. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  26. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  27. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  28. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  29. 2021How to Train BERT with an Academic Budgetcited by 80
  30. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  31. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  32. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  33. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  34. 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 18
  35. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 79; 2025: 77; 2026: 80.

260 papers.

Sort:

By area:

  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2025Muon is Scalable for LLM Trainingcited by 381
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  9. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  10. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  11. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  12. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
38 more
  1. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  2. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  3. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  4. 2024A New Perspective on Shampoo's Preconditionercited by 54
  5. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  6. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  7. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  8. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  9. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  10. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  11. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  12. 2025In Search of Adam's Secret Saucecited by 40
  13. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  14. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  15. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  16. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  17. 2024A Survey of Small Language Modelscited by 31
  18. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  19. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  20. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  21. 2025On the Convergence of Muon and Beyondcited by 28
  22. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  23. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  24. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  25. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  26. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  27. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  28. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  29. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  30. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  31. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  32. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  33. 2024Transformers Struggle to Learn to Searchcited by 19
  34. 2023Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomnesscited by 18
  35. 2025Tool Unlearning for Tool-Augmented LLMscited by 17
  36. 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
  37. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 16
  38. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15

50 most cited of 260 All 260 (JSON)