# Optimization Hyper-parameter Laws for Large Language Models

arXiv: https://arxiv.org/abs/2409.04777 · HTML: https://mlsanity.com/paper/2409.04777/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2024.

## Built on by (most cited first, 7 of 7)

- [A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules](https://mlsanity.com/paper/2503.12811/) (2025)
- [Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation](https://mlsanity.com/paper/2411.04358/) (2024)
- [Learning Dynamics in Continual Pre-Training for Large Language Models](https://mlsanity.com/paper/2505.07796/) (2025)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability](https://mlsanity.com/paper/2603.12038/) (2026)
- [Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2509.07238/) (2025)
- [Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference](https://mlsanity.com/paper/2610.14277/) (2026)

## Abstract (arXiv)

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.