mlsanityData to 7 Oct 2026

Scalable Optimization in the Modular Norm

arXiv 2405.14813 ↗First posted 23 May 2024cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the width of a single layer, graceful scaling of training has been linked to the need to normalize the weights and their updates in the "natural norm" particular to that layer. In this paper, we significantly generalize this idea by defining the modular norm, which is the natural norm on the full weight space of any neural network architecture. The modular norm is defined recursively in tandem with the network architecture itself. We show that the modular norm has several promising applications. On the practical side, the modular norm can be used to normalize the updates of any base optimizer so that the learning rate becomes transferable across width and depth. This means that the user does not need to compute optimizer-specific scale factors in order to scale training. On the theoretical side, we show that for any neural network built from "well-behaved" atomic modules, the gradient of the network is Lipschitz-continuous in the modular norm, with the Lipschitz constant admitting a simple recursive formula. This characterization opens the door to porting standard ideas in optimization theory over to deep learning. We have created a Python package called Modula that automatically normalizes weight updates in the modular norm of the architecture. The package is available via "pip install modula" with source code at https://github.com/jxbz/modula.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

32 of 47 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2017Deep Neural Networks as Gaussian Processescited by 617
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  12. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
20 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  2. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  3. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  4. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  5. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  6. 2023A Spectral Condition for Feature Learningcited by 101
  7. 2018Measuring and regularizing networks in function spacecited by 97
  8. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  9. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  10. 2016Theano: A Python framework for fast computation of mathematical expressionscited by 64
  11. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  12. 2020On the distance between two neural networks and the stability of learningcited by 46
  13. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  14. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  15. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  16. 2023Depth Dependence of μP Learning Rates in ReLU MLPscited by 9
  17. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6
  18. 2022Automatically Bounding the Taylor Remainder Series: Tighter Bounds and New Applicationscited by 5
  19. 2023Universal Majorization-Minimization Algorithmscited by 3
  20. 2023Automatic Optimisation of Normalised Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 24; 2026: 34.

64 papers.

Sort:

By area:

  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2024Old Optimizer, New Norm: An Anthologycited by 173
  3. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  4. 2024Modular Duality in Deep Learningcited by 71
  5. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  6. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  7. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  8. 2025Dion: Distributed Orthonormalized Updatescited by 64
  9. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  10. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  11. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 25
  12. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
38 more
  1. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  2. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  3. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  4. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  5. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  6. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  7. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  8. 2025Dion2: A Simple Method to Shrink Matrix in Muoncited by 11
  9. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  10. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  11. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 8
  12. 2025Physics of Skill Learningcited by 8
  13. 2025FOCUS: First Order Concentrated Updating Schemecited by 7
  14. 2025Beyond Outliers: A Study of Optimizers Under Quantizationcited by 6
  15. 2026TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizerscited by 6
  16. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 6
  17. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  18. 2026Muown: Row-Norm Control for Muon Optimizationcited by 6
  19. 2025Optimal Scaling Needs Optimal Normcited by 5
  20. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  21. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 5
  22. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  23. 2025Function-Space Learning Ratescited by 3
  24. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  25. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 3
  26. 2026LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifoldcited by 3
  27. 2026SimpleGPT: Improving GPT via A Simple Normalization Strategycited by 2
  28. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  29. 2026Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noisecited by 2
  30. 2025Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updatescited by 1
  31. 2026Learning Rate Transfer in Normalized Transformerscited by 1
  32. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  33. 2026Error whitening: Why Gauss-Newton outperforms Newtoncited by 1
  34. 2026Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimizationcited by 1
  35. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  36. 2026Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvaturecited by 1
  37. 2026Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muoncited by 1
  38. 2025Follow the MEP: Scalable Neural Representations for Minimum-Energy Path Discovery in Molecular Systemscited by 0

50 most cited of 64 All 64 (JSON)