mlsanityData to 14 Oct 2026

How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

arXiv 2609.39767 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The scale and expense of pre-training language models make efficient hyperparameter tuning essential, yet a principled guidance is still missing. In this work, we analyze language model pre-training dynamics from a local landscape geometry perspective. Our study reveals two distinct phases. In Phase I, sharpness of the local landscape is initially high, leading to instability and loss plateaus under large learning rates (LRs). The landscape shifts from sharp to flatter regions early in training. This dynamic explains the necessity of LR warmup and further suggests that larger peak LRs require proportionally longer warmup periods. In Phase II, the local landscape is governed by the gradient noise scale. Our theory identifies a depth flatness trade-off: high noise from smaller batches widens the loss basin, whereas reduced noise from larger batches deepens it. This theory motivates a dynamic batch-size (BS) scheduler that begins with a small BS and increases it late in training. Together, we provide a unified view of loss landscape evolution, which translates into actionable tuning strategies for large-scale pre-training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

43 of 58 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
31 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  7. 2018An Empirical Model of Large-Batch Trainingcited by 281
  8. 2017Three Factors Influencing Minima in SGDcited by 273
  9. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  10. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  11. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  12. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  13. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  14. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  15. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  16. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  17. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  18. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  19. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  20. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
  21. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  22. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  23. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 26
  24. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  25. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
  26. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  27. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  28. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  29. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  30. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
  31. 2024AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Sizecited by 3

Built on by

Not cited yet.