mlsanityData to 7 Oct 2026

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

arXiv 2203.03466 ↗First posted 7 Mar 2022cs.LG cond-mat.dis-nn cs.NEcited by 272

Not on the map. Why?

Abstract (arXiv)

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`.

Key facts. Cited by 272 papers in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

53 of 69 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016Wide Residual Networkscited by 3,057
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
38 more
  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  7. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2017Deep Neural Networks as Gaussian Processescited by 617
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2021Pay Attention to MLPscited by 367
  12. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  13. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  14. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  15. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  16. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  17. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  18. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  19. 2018An Empirical Model of Large-Batch Trainingcited by 268
  20. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  21. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  22. 2020Understanding the Difficulty of Training Transformerscited by 210
  23. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  24. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  25. 2016Deep Information Propagationcited by 194
  26. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  27. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  28. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  29. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  30. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  31. 2018Training Tips for the Transformer Modelcited by 85
  32. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  33. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  34. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  35. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  36. 2020On the distance between two neural networks and the stability of learningcited by 46
  37. 2021RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognitioncited by 40
  38. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 40; 2024: 61; 2025: 75; 2026: 79.

272 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2023Large Language Modelscited by 774
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  10. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  12. 2022Dataset Distillation using Neural Feature Regressioncited by 177
38 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  3. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  4. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  5. 2023Likelihood-Based Diffusion Language Modelscited by 147
  6. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  7. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  8. 2024Barren Plateaus in Variational Quantum Computingcited by 130
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  10. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  11. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  12. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  13. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  14. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  15. 2023A Spectral Condition for Feature Learningcited by 101
  16. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  17. 2024Scaling Laws for Precisioncited by 92
  18. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  19. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  20. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  21. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  22. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  23. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  24. 2025Distillation Scaling Lawscited by 59
  25. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  26. 2025Horizon Reduction Makes RL Scalablecited by 55
  27. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  28. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  29. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  30. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  31. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  32. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  33. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  34. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  35. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  36. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  37. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  38. 2024Memory3: Language Modeling with Explicit Memorycited by 39

50 most cited of 272 All 272 (JSON)