mlsanityData to 7 Oct 2026

Efficient Hyperparameter Tuning via Trajectory Invariance Principle

arXiv 2509.25049 ↗First posted 29 Sep 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

As hyperparameter tuning becomes increasingly costly at scale, efficient tuning methods are essential. Yet principles for guiding hyperparameter tuning remain limited. In this work, we seek to establish such principles by considering a broad range of hyperparameters, including batch size, learning rate, and weight decay. We identify a phenomenon we call trajectory invariance, where pre-training loss curves, gradient noise, and gradient norm exhibit invariance--closely overlapping--with respect to a quantity that combines learning rate and weight decay. This phenomenon effectively reduces the original two-dimensional hyperparameter space to one dimension, yielding an efficient tuning rule: follow the salient direction revealed by trajectory invariance. Furthermore, we refine previous scaling laws and challenge several existing viewpoints. Overall, our work proposes new principles for efficient tuning and inspires future research on scaling laws.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

30 of 37 references.

References · 30

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  8. 2018An Empirical Model of Large-Batch Trainingcited by 268
  9. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  10. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  11. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  12. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
18 more
  1. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  4. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 52
  5. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  6. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  7. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  8. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  9. 2024Scaling Optimal LR Across Token Horizonscited by 35
  10. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  11. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  12. 2024Scaling Law with Learning Rate Annealingcited by 29
  13. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 24
  14. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  15. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  16. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 21
  17. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 20
  18. 2024Scaling Law for Language Models Training Considering Batch Sizecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  2. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  3. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 2
  4. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0