mlsanityData to 14 Oct 2026

Modular Duality in Deep Learning

arXiv 2410.21265 ↗First posted 28 Oct 2024cs.LG cs.NE stat.MLcited by 77

Not on the map. Why?

Abstract (arXiv)

An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the weights reside. We take this idea seriously in this paper and construct such a duality map for general neural networks. Our map, which we call modular dualization, forms a unifying theoretical basis for training algorithms that are a) fast and b) scalable. Modular dualization involves first assigning operator norms to layers based on the semantics of each layer, and then using these layerwise norms to recursively induce a duality map on the weight space of the full neural architecture. We conclude by deriving GPU-friendly algorithms for dualizing Embed, Linear and Conv2D layers -- the latter two methods are based on a rectangular Newton-Schulz iteration (Kovarik, 1970; Bj"orck & Bowie, 1971). A variant of our methods was used to set speed records for training NanoGPT. Overall, we hope that our theory of modular duality will yield a next generation of fast and scalable optimizers for general neural architectures.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 15 of its 36 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

15 of 36 references.

References · 15

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  4. 2024Old Optimizer, New Norm: An Anthologycited by 193
  5. 2023A Spectral Condition for Feature Learningcited by 109
  6. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  7. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  8. 2024Scalable Optimization in the Modular Normcited by 68
  9. 2023Benchmarking Neural Network Training Algorithmscited by 52
  10. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17
  11. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  12. 2017The duality structure gradient descent algorithm: analysis and applications to neural networkscited by 8
3 more
  1. 2022Automatically Bounding the Taylor Remainder Series: Tighter Bounds and New Applicationscited by 5
  2. 2020Effect of the initial configuration of weights on the training and function of artificial neural networkscited by 4
  3. 2023Universal Majorization-Minimization Algorithmscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 51.

77 papers.

Sort:

By area:

  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  2. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  3. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  4. 2025Dion: Distributed Orthonormalized Updatescited by 68
  5. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  6. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  7. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  8. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  9. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  10. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  11. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 28
  12. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
38 more
  1. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  2. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 16
  3. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 13
  4. 2025Dion2: A Simple Method to Shrink Matrix in Muoncited by 13
  5. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  6. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  7. 2025Effective Quantization of Muon Optimizer Statescited by 11
  8. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
  9. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  10. 2025Optimal Scaling Needs Optimal Normcited by 8
  11. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  12. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 7
  13. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  14. 2026TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizerscited by 6
  15. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 6
  16. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  17. 2026Muown: Row-Norm Control for Muon Optimizationcited by 6
  18. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  19. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  20. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  21. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  22. 2026GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generationcited by 5
  23. 2026Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learningcited by 5
  24. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  25. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  26. 2026Orthogonal Model Mergingcited by 4
  27. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 4
  28. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 4
  29. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  30. 2026Spectral Scaling Laws of Muoncited by 4
  31. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  32. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  33. 2025ENSAM: an efficient foundation model for interactive segmentation of 3D medical imagescited by 2
  34. 2026Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descentcited by 2
  35. 2026NS-RGS: Newton-Schulz based Riemannian gradient method for orthogonal group synchronizationcited by 2
  36. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  37. 2026When to use what Schatten-p norm in deep learning?cited by 2
  38. 2026Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networkscited by 2

50 most cited of 77 All 77 (JSON)