# Optimizers and stability in LLM pre-training

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/optimizers-and-stability-in-llm-pre-training/

## Key facts

- Optimizers and stability in LLM pre-training: 113 papers on arXiv since 2021 (data through 19 Oct 2026).
- 37 of its 113 mapped papers (33%) cite “Training Compute-Optimal Large Language Models” (2022, arXiv 2203.15556).
- 20 of its 113 mapped papers (18%) cite “DeepSeek-V3 Technical Report” (2024, arXiv 2412.19437).
- 6 new papers 1 Oct–19 Oct.

## Most cited here

- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022), cited by 37 papers here
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024), cited by 20 papers here
- [Spike No More: Stabilizing the Pre-training of Large Language Models](https://mlsanity.com/paper/2312.16903/) (2023), cited by 13 papers here
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023), cited by 12 papers here
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023), cited by 12 papers here
- [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://mlsanity.com/paper/2410.05192/) (2024), cited by 10 papers here
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022), cited by 8 papers here
- [CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation](https://mlsanity.com/paper/2502.10940/) (2025), cited by 6 papers here
- [The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training](https://mlsanity.com/paper/2502.19002/) (2025), cited by 6 papers here
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023), cited by 4 papers here

## New (1 Oct–19 Oct)

- [T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training](https://mlsanity.com/paper/2610.24938/) (2026-10-18)
- [CAST: History-Adaptive Spectral Clipping for LLM Pretraining](https://mlsanity.com/paper/2610.19761/) (2026-10-14)
- [Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models](https://mlsanity.com/paper/2610.13524/) (2026-10-09)
- [Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design](https://mlsanity.com/paper/2610.04867/) (2026-10-04)
- [OptiSelect: How does the Optimizer Shape Data Curriculum?](https://mlsanity.com/paper/2610.03432/) (2026-10-02)
- [Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance](https://mlsanity.com/paper/2610.02355/) (2026-10-01)