# Dion2: A Simple Method to Shrink Matrix in Muon

arXiv: https://arxiv.org/abs/2512.16928 · HTML: https://mlsanity.com/paper/2512.16928/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 12 of 12)

- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [Modular Duality in Deep Learning](https://mlsanity.com/paper/2410.21265/) (2024)
- [Scalable Optimization in the Modular Norm](https://mlsanity.com/paper/2405.14813/) (2024)
- [Dion: Distributed Orthonormalized Updates](https://mlsanity.com/paper/2504.05295/) (2025)
- [MuonBP: Faster Muon via Block-Periodic Orthogonalization](https://mlsanity.com/paper/2510.16981/) (2025)
- [FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models](https://mlsanity.com/paper/2505.17967/) (2025)

## Built on by (most cited first, 13 of 13)

- [The Newton-Muon Optimizer](https://mlsanity.com/paper/2604.01472/) (2026)
- [Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization](https://mlsanity.com/paper/2601.19400/) (2026)
- [Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less](https://mlsanity.com/paper/2605.06654/) (2026)
- [Spectral Scaling Laws of Muon](https://mlsanity.com/paper/2606.04058/) (2026)
- [Anytime Training with Schedule-Free Spectral Optimization](https://mlsanity.com/paper/2605.23061/) (2026)
- [Tight Complexity of Muon under Heavy-Tailed Noise](https://mlsanity.com/paper/2605.18528/) (2026)
- [Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization](https://mlsanity.com/paper/2606.27216/) (2026)
- [Scale Weight Decay and Train Better](https://mlsanity.com/paper/2607.23777/) (2026)
- [When and Why Grouping Attention Heads Accelerates Muon Optimization](https://mlsanity.com/paper/2605.08933/) (2026)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Scaling Muon for Diffusion Transformers](https://mlsanity.com/paper/2608.20818/) (2026)
- [Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization](https://mlsanity.com/paper/2605.16341/) (2026)
- [MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads](https://mlsanity.com/paper/2610.02705/) (2026)

## Abstract (arXiv)

The Muon optimizer enjoys strong empirical performance and theoretical grounding. However, the super-linear cost of its orthonormalization step introduces increasing overhead with scale. To alleviate this cost, several works have attempted to reduce the size of the matrix entering the orthonormalization step. We introduce Dion2, a much simpler method for shrinking the matrix involved in Muon's computation compared to prior approaches. At a high level, Dion2 selects a fraction of rows or columns at each iteration and orthonormalizes only those. This sampling procedure makes the update sparse, reducing both computation and communication costs which in turn improves the scalability of Muon.