mlsanityData to 7 Oct 2026

μLO: Compute-Efficient Meta-Generalization of Learned Optimizers

arXiv 2406.00153 ↗First posted 31 May 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (meta-generalize), especially when training networks wider than those seen during meta-training. To address this, we derive the Maximal Update Parametrization (μP) for two state-of-the-art learned optimizer architectures and propose a simple meta-training recipe for μ-parameterized LOs (μLOs). Our empirical evaluation demonstrates that LOs meta-trained with our recipe substantially improve meta-generalization to wider unseen tasks when compared to LOs trained under standard parametrization (SP) using the same compute budget. We also empirically observe that μLOs exhibit unexpectedly improved meta-generalization to deeper networks (5× meta-training) and surprising generalization to much longer training horizons (25× meta-training) when compared to SP LOs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

32 of 52 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Root Mean Square Layer Normalizationcited by 1,425
  8. 2016Learning to learn by gradient descent by gradient descentcited by 616
  9. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  11. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  12. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
20 more
  1. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  2. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  3. 2017Learned Optimizers that Scale and Generalizecited by 109
  4. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  5. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  6. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  7. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  8. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  9. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  10. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  11. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 35
  12. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  13. 2020Training Stronger Baselines for Learning to Optimizecited by 31
  14. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  15. 2021A Generalizable Approach to Learning Optimizerscited by 23
  16. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  17. 2022A Simple Guard for Learned Optimizerscited by 9
  18. 2023Learning to Generalize Provably in Learning to Optimizecited by 8
  19. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  20. 2023Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  2. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  3. 2025PyLO: Towards Accessible Learned Optimizers in PyTorchcited by 1
  4. 2026Celo2: Towards Learned Optimization Free Lunchcited by 1
  5. 2026Extending μP: Spectral Conditions for Feature Learning Across Optimizerscited by 0
  6. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  7. 2026Learning to Optimize Radiotherapy Plans via Fluence Maps Diffusion Model Generation and LSTM-based Optimizationcited by 0