mlsanityData to 7 Oct 2026

OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biases

arXiv 2602.01105 ↗First posted 1 Feb 2026cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce nameA (fullname), which combines spectral control from orthogonalized update directions with ℓ_∞-style coordinate control from sign updates. nameA forms a Lion-style momentum direction, approximately orthogonalizes it via a few Newton--Schulz iterations, and then applies an entrywise sign, providing an efficient approximation to taking a maximal step over the intersection of the spectral and ℓ_∞ constraint sets (a scaled Hadamard-like set for matrix parameters). Despite the strong nonlinearity of orthogonalization and sign, we prove convergence under a mild, empirically verified diagonal-isotropy assumption. Across large-scale language and vision training, including GPT-2 and Llama pretraining, SiT image pretraining, and supervised fine-tuning, nameA matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level optimizer state, and it mitigates optimizer mismatch when fine-tuning AdamW-pretrained checkpoints.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

36 of 55 references.

References · 36

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  7. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  9. 2025Muon is Scalable for LLM Trainingcited by 381
  10. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  11. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
24 more
  1. 2024Old Optimizer, New Norm: An Anthologycited by 173
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  3. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  4. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 80
  5. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  6. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  7. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  8. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  9. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  10. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  11. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  12. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  13. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 34
  14. 2024The Implicit Bias of Adam on Separable Datacited by 32
  15. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  16. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  17. 2025On the Convergence of Muon and Beyondcited by 28
  18. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 25
  19. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  20. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  21. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  22. 2025Effective Quantization of Muon Optimizer Statescited by 10
  23. 2025Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settingscited by 6
  24. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2
  2. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 0
  3. 2026Physical Muon: Orthogonalization as an Equilibrium Computationcited by 0