# An Empirical Study of μP Learning Rate Transfer

arXiv: https://arxiv.org/abs/2404.05728 · HTML: https://mlsanity.com/paper/2404.05728/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2024.

## Built on by (most cited first, 10 of 10)

- [Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling](https://mlsanity.com/paper/2409.15156/) (2024)
- [Understanding the Mechanisms of Fast Hyperparameter Transfer](https://mlsanity.com/paper/2512.22768/) (2025)
- [Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size](https://mlsanity.com/paper/2506.15025/) (2025)
- [Pre-Training LLMs on a budget: A comparison of three optimizers](https://mlsanity.com/paper/2507.08472/) (2025)
- [A Proof of Learning Rate Transfer under μP](https://mlsanity.com/paper/2511.01734/) (2025)
- [Learning Rate Transfer in Normalized Transformers](https://mlsanity.com/paper/2604.27077/) (2026)
- [Statistical Properties of Training & Generalization](https://mlsanity.com/paper/2606.20299/) (2026)
- [Relative Generalization Invariance of LLM Pretraining](https://mlsanity.com/paper/2609.33016/) (2026)
- [Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons](https://mlsanity.com/paper/2609.35029/) (2026)
- [Learning Rate Transfer for Hybrid Transformer-SSM Architectures](https://mlsanity.com/paper/2610.01172/) (2026)

## Abstract (arXiv)

Deep learning models have become a cornerstone of modern AI research, yet their initializations and learning rates may at times be set in an opaque or ad-hoc fashion due to the high cost of hyperparameter sweeps. The μ-Parameterization (μP) offers a possible solution to this challenge, yielding scaling rules for model initialization and learning rates while reportedly enabling zero-shot hyperparameter transfer from small to large models. Despite its evident promise, the μP method is not yet widely adopted, perhaps due to higher implementation complexity, many variations, or complex theoretical background. This work considers μP empirically, focusing on the popular transformer architecture, and aims to answer a simple question: does μ-Transfer yield near-optimal learning rates in practice? Studying over a dozen ablations with up to 1.2B parameters and 33B tokens and a large-scale experiment with up to 10B parameters and 190B tokens, we observe a positive answer for most settings, and discuss improvements otherwise.