mlsanityData to 14 Oct 2026

ASGO: Adaptive Structured Gradient Optimization

arXiv 2503.20762 ↗First posted 26 Mar 2025cs.LG math.OCcited by 57

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this structural representation, it has been widely observed that gradients are low-rank and Hessians are approximately block diagonal. These structured properties are crucial for designing efficient optimization algorithms, but are not utilized by many current popular optimizers like Adam. In this paper, we present a novel optimization algorithm ASGO that capitalizes on these properties by employing a preconditioner that is adaptively updated using structured gradients. By a fine-grained theoretical analysis, ASGO is proven to achieve superior convergence rates compared to existing structured gradient methods. Based on this convergence theory, we further demonstrate that ASGO can benefit from low-rank gradients and block diagonal Hessians. We also discuss practical modifications of ASGO and empirically verify ASGO's effectiveness on language model tasks. Code is available at https://github.com/infinity-stars/ASGO.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

54 of 80 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2025Muon is Scalable for LLM Trainingcited by 428
38 more
  1. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  2. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  3. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  4. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  5. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  6. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  7. 2024Old Optimizer, New Norm: An Anthologycited by 193
  8. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  9. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  10. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  11. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  13. 2020Momentum Improves Normalized SGDcited by 139
  14. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  15. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  16. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  17. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  18. 2023A Spectral Condition for Feature Learningcited by 109
  19. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  20. 2025A Note on the Convergence of Muoncited by 83
  21. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  22. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  23. 2024Modular Duality in Deep Learningcited by 77
  24. 2024Scalable Optimization in the Modular Normcited by 68
  25. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  26. 2024A New Perspective on Shampoo's Preconditionercited by 59
  27. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  28. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  29. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  30. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  31. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  32. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  33. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  34. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  35. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  36. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  37. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  38. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 35.

57 papers.

Sort:

  1. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  2. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  3. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  4. 2026Convergence of Muon with Newton-Schulzcited by 36
  5. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  6. 2025On the Convergence of Muon and Beyondcited by 31
  7. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  8. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  9. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  10. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  11. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 22
  12. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
38 more
  1. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 17
  2. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  3. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  4. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  5. 2026Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimizationcited by 14
  6. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  7. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  8. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  9. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  10. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 11
  11. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  12. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  13. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  14. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  15. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  16. 2025Unbiased Gradient Low-Rank Projectioncited by 7
  17. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  18. 2025On the Convergence Analysis of Muoncited by 5
  19. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 5
  20. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  21. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  22. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  23. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  24. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 4
  25. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  26. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  27. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  28. 2025Federated Stochastic Minimax Optimization under Heavy-Tailed Noisescited by 2
  29. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  30. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  31. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  32. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2
  33. 2026Scale Weight Decay and Train Bettercited by 2
  34. 2026Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioningcited by 1
  35. 2026Optimal Projection-Free Adaptive SGD for Matrix Optimizationcited by 1
  36. 2026Lightweight Geometric Adaptation for Training Physics-Informed Neural Networkscited by 1
  37. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  38. 2026Schattor: Schatten-family methods for deep learning optimizationcited by 1

50 most cited of 57 All 57 (JSON)