mlsanityData to 14 Oct 2026

Understanding the Mechanisms of Fast Hyperparameter Transfer

arXiv 2512.22768 ↗First posted 28 Dec 2025cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparameters, which can enable direct transfer of optimal HPs from small-scale grid searches to large models with minimal performance loss. To understand the principles governing such transfer strategy, we develop a general conceptual framework for reasoning about HP transfer across scale, characterizing transfer as fast when the suboptimality it induces vanishes asymptotically faster than the finite-scale performance gap. We show formally that fast transfer is equivalent to useful transfer for compute-optimal grid search, meaning that transfer is asymptotically more compute-efficient than direct tuning. While empirical work has found that the Maximal Update Parameterization (μP) exhibits fast transfer when scaling model width, the mechanisms remain poorly understood. We show that this property depends critically on problem structure by presenting synthetic settings where transfer either offers provable computational advantage or fails to outperform direct tuning even under μP. To explain the fast transfer observed in practice, we conjecture that decomposing the optimization trajectory reveals two contributions to loss reduction: (1) a width-stable component that determines the optimal HPs, and (2) a width-sensitive component that improves with width but weakly perturbs the HP optimum. We present empirical evidence for this hypothesis across various settings, including large language model pretraining.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 28 Dec 2025.

Builds on

42 of 56 references.

References · 42

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  9. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  10. 2021Deep learning: a statistical viewpointcited by 240
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  12. 2022Datamodels: Predicting Predictions from Training Datacited by 189
30 more
  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  8. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  9. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  10. 2025Dion: Distributed Orthonormalized Updatescited by 68
  11. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  12. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  13. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  14. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  15. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  16. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  17. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 38
  18. 2024Does SGD really happen in tiny subspaces?cited by 36
  19. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
  20. 2025When do spectral gradient updates help in deep learning?cited by 31
  21. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 26
  22. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  23. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  24. 2021Optimization-Based Separations for Neural Networkscited by 17
  25. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 17
  26. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  27. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
  28. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  29. 2024An Empirical Study of μP Learning Rate Transfercited by 10
  30. 2025Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  3. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  4. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  5. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 3
  6. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  7. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0