mlsanityData to 19 Oct 2026

Optimizers and stability in LLM pre-training

113 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: training, large, pre-training, language, loss

Key facts. Optimizers and stability in LLM pre-training: 113 papers on arXiv since 2021 (data through 19 Oct 2026). 37 of its 113 mapped papers (33%) cite “Training Compute-Optimal Large Language Models” (2022, arXiv 2203.15556). 6 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

6 new papers, 1 Oct – 19 Oct; 0 this week.

  1. Earlier · 6
  2. T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
  3. CAST: History-Adaptive Spectral Clipping for LLM Pretraining
  4. Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models
  5. Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design
  6. OptiSelect: How does the Optimizer Shape Data Curriculum? also fits: Data mixtures and data selection
  7. Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance

Foundations

Most cited by papers here

  1. Training Compute-Optimal Large Language Models2022cited by 37 here
  2. DeepSeek-V3 Technical Report2024cited by 20 here
  3. Spike No More: Stabilizing the Pre-training of Large Language Models2023cited by 13 here
  4. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training2023cited by 12 here
  5. Small-scale proxies for large-scale Transformer training instabilities2023cited by 12 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020Scaling Laws for Neural Language Modelscited by 38 here
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 8 here
  3. 2022Training Compute-Optimal Large Language Modelscited by 37 here
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 12 here
  5. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 4 here
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 3 here
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 17 here
  8. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 9 here
16 more it builds on, not in this chain
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 13 here
  2. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 3 here
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 8 here
  4. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 4 here
  5. 2023Loss Spike in Training Neural Networkscited by 4 here
  6. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 12 here
  7. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 3 here
  8. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 12 here
  9. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 13 here
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 11 here
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 7 here
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 21 here
  13. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 10 here
  14. 2024DeepSeek-V3 Technical Reportcited by 20 here
  15. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 6 here
  16. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 6 here

The 24 most cited of 80.

Browse

30 of 113, ranked 6–35

  1. 6Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective202410
  2. 7Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer20228
  3. 8CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation20256
  4. 9The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training20256
  5. 10Masked Structural Growth for 2x Faster Language Model Pre-training20234
  6. 11Loss Spike in Training Neural Networks20234
  7. 12GradPower: Powering Gradients for Faster Language Model Pre-Training20253
  8. 13Overcoming the Communication-Performance Tradeoff in LLM Pretraining20253
  9. 14Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training20243
  10. 15TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training20243
All 30
  1. 16Early Weight Averaging meets High Learning Rates for LLM Pre-training20233
  2. 17Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning20233
  3. 18The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models20213
  4. 19Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging20262
  5. 20Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement20262
  6. 21ZClip: Adaptive Spike Mitigation for LLM Pre-Training20252
  7. 22GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection20252
  8. 23Recipes for Pre-training LLMs with MXFP820252
  9. 24Memory-Efficient LLM Pretraining via Minimalist Optimizer Design20252
  10. 25Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler20252
  11. 26Benchmarking Optimizers for Large Language Model Pretraining20252
  12. 27Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes20242
  13. 28Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers20261
  14. 29Preconditioning Benefits of Spectral Orthogonalization in Muon20261
  15. 30TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training20261
  16. 31Stabilizing Native Low-Rank LLM Pretraining20261
  17. 32Demystifying Manifold Constraints in LLM Pre-training20261
  18. 33Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay20261
  19. 34Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining20261
  20. 35Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining20251