# Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization

arXiv: https://arxiv.org/abs/2606.27216 · HTML: https://mlsanity.com/paper/2606.27216/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Fantastic Pretraining Optimizers and Where to Find Them](https://mlsanity.com/paper/2509.02046/) (2025)
- [NorMuon: Making Muon more efficient and scalable](https://mlsanity.com/paper/2510.05491/) (2025)
- [Benchmarking Optimizers for Large Language Model Pretraining](https://mlsanity.com/paper/2509.01440/) (2025)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [Convergence of Muon with Newton-Schulz](https://mlsanity.com/paper/2601.19156/) (2026)
- [MuonBP: Faster Muon via Block-Periodic Orthogonalization](https://mlsanity.com/paper/2510.16981/) (2025)
- [When do spectral gradient updates help in deep learning?](https://mlsanity.com/paper/2512.04299/) (2025)
- [MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration](https://mlsanity.com/paper/2603.28254/) (2026)
- [RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization](https://mlsanity.com/paper/2603.20527/) (2026)
- [Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory](https://mlsanity.com/paper/2603.26554/) (2026)
- [Muon is Not That Special: Random or Inverted Spectra Work Just as Well](https://mlsanity.com/paper/2605.11181/) (2026)
- [Dion2: A Simple Method to Shrink Matrix in Muon](https://mlsanity.com/paper/2512.16928/) (2025)
- [Hierarchical Roofline Performance Analysis for Deep Learning Applications](https://mlsanity.com/paper/2009.05257/) (2020)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training](https://mlsanity.com/paper/2601.23261/) (2026)
- [TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers](https://mlsanity.com/paper/2602.13498/) (2026)
- [MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training](https://mlsanity.com/paper/2602.21545/) (2026)
- [Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training](https://mlsanity.com/paper/2603.17970/) (2026)
- [When and Why Grouping Attention Heads Accelerates Muon Optimization](https://mlsanity.com/paper/2605.08933/) (2026)

## Built on by (most cited first, 2 of 2)

- [Dion3: Full-Stack Orthogonal Updates](https://mlsanity.com/paper/2608.11612/) (2026)
- [Scaling Muon for Diffusion Transformers](https://mlsanity.com/paper/2608.20818/) (2026)

## Abstract (arXiv)

Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an H × W matrix, with r=minH,W and s=maxH,W, K steps of the full-matrix Newton-Schulz update require O(r2 s K) work and couple all rows and columns through repeated Gram matrix products. We introduce Hierarchical Muon (HiMuon), a tiled Newton-Schulz scheme for Muon-type optimization. HiMuon partitions each momentum-gradient matrix into T × T tiles, applies the same finite Newton-Schulz map independently to each tile, and reassembles the results. For finite T below the matrix dimensions, HiMuon defines a local matrix-function map rather than a convergent approximation to the full-matrix update: spectral interactions are preserved within tiles and discarded across tile boundaries. For fixed finite T, the leading Newton-Schulz work decreases to O(H W T K), and the computation decomposes into independent small dense matrix operations. This structure enables tile-size-dependent GPU kernels, cross-layer batching, memory-bounded chunking, and runtime tile-size schedules. Experiments on transformer training and controlled matrix-function diagnostics show that HiMuon improves optimizer-step efficiency while keeping training behavior close to full-matrix Muon in the tested regimes.