mlsanityData to 14 Oct 2026

Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

arXiv 2605.12492 ↗First posted 12 May 2026cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 69 of its 87 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

69 of 87 references.

References · 69

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2016Layer Normalizationcited by 5,669
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  13. 2020On Layer Normalization in the Transformer Architecturecited by 922
  14. 2019On the Convergence of Adam and Beyondcited by 841
  15. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  16. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  17. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  18. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  19. 2025Kimi K2: Open Agentic Intelligencecited by 567
  20. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  21. 2025Muon is Scalable for LLM Trainingcited by 428
  22. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  23. 2024LoRA Learns Less and Forgets Lesscited by 328
  24. 2020Query-Key Normalization for Transformerscited by 280
  25. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  26. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  27. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  28. 2018Riemannian Adaptive Optimization Methodscited by 194
  29. 2024Old Optimizer, New Norm: An Anthologycited by 193
  30. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  31. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  32. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  33. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  34. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  35. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  36. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  37. 2023A Spectral Condition for Feature Learningcited by 109
  38. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filteringcited by 2
  2. 2026ISO: An RLVR-Native Optimization Stackcited by 2
  3. 2026PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Trainingcited by 1
  4. 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oraclescited by 0
  5. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Trainingcited by 0
  6. 2026When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?cited by 0