mlsanityData to 7 Oct 2026

Dion: Distributed Orthonormalized Updates

arXiv 2504.05295 ↗First posted 7 Apr 2025cs.LG cs.AI math.OCcited by 64

Not on the map. Why?

Abstract (arXiv)

Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash with sharded weights in large-scale LLM training, causing high compute and communication cost. We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule that replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer, avoiding full-matrix reconstruction and integrating cleanly with weight sharding. The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings. On language models from 160M to 3B parameters, Dion retains the benefits of orthonormalized updates, while markedly reducing wall-clock time at scale, making it a practical optimizer for next-generation foundation models. Code is available at: https://github.com/microsoft/dion/

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

35 of 55 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2016Layer Normalizationcited by 5,579
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
23 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  7. 2025Muon is Scalable for LLM Trainingcited by 381
  8. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  9. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  10. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  11. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  12. 2024Old Optimizer, New Norm: An Anthologycited by 173
  13. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  14. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  15. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  16. 2023A Spectral Condition for Feature Learningcited by 101
  17. 2024Modular Duality in Deep Learningcited by 71
  18. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  19. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  20. 2024Scalable Optimization in the Modular Normcited by 64
  21. 2024Does SGD really happen in tiny subspaces?cited by 35
  22. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  23. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 44.

64 papers.

Sort:

By area:

  1. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  2. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  3. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  4. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  5. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  6. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  7. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  8. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  9. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  10. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 15
  11. 2025Dion2: A Simple Method to Shrink Matrix in Muoncited by 11
  12. 2026Insights on Muon from Simple Quadraticscited by 11
38 more
  1. 2025Effective Quantization of Muon Optimizer Statescited by 10
  2. 2026Arcee Trinity Large Technical Reportcited by 10
  3. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 8
  4. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  5. 2025Spherical Leech Quantization for Visual Tokenization and Generationcited by 6
  6. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  7. 2025Optimal Scaling Needs Optimal Normcited by 5
  8. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  9. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 5
  10. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  11. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  12. 2026PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Trainingcited by 4
  13. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 4
  14. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  15. 2025A Proof of Learning Rate Transfer under μPcited by 3
  16. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  17. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 3
  18. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  19. 2026CacheMuon: Using Temporal Preconditioning To Approximate Polar Factorcited by 3
  20. 2026Dion3: Full-Stack Orthogonal Updatescited by 3
  21. 2025FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Modelscited by 2
  22. 2026Learning to Advect: A Neural Semi-Lagrangian Architecture for Weather Forecastingcited by 2
  23. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2
  24. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  25. 2026Spectral Scaling Laws of Muoncited by 2
  26. 2026Scale Weight Decay and Train Bettercited by 2
  27. 2025Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updatescited by 1
  28. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  29. 2026When and Why Grouping Attention Heads Accelerates Muon Optimizationcited by 1
  30. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 1
  31. 2026Parallax: Parameterized Local Linear Attention for Language Modelingcited by 1
  32. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  33. 2026DMuon: Efficient Distributed Muon Training with Near-Adam Overheadcited by 1
  34. 2026Learning to Discover Iterative Spectral Algorithmscited by 0
  35. 2026Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iterationcited by 0
  36. 2026MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimizationcited by 0
  37. 2026Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectivescited by 0
  38. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0

50 most cited of 64 All 64 (JSON)