mlsanityData to 14 Oct 2026

PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descent

arXiv 2605.10335 ↗First posted 11 May 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by ℓp-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an O(1/√T) term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving textttfp32 optimizer-state memory relative to AdamW. Combined with uniform textttint8 quantization, PowerStep remains numerically stable and reduces optimizer-state memory by ~8× compared to textttfp32 AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

40 of 57 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Convex optimizationcited by 2,515
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2019On the Convergence of Adam and Beyondcited by 841
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 434
28 more
  1. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  2. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2024Old Optimizer, New Norm: An Anthologycited by 193
  6. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  7. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  8. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  9. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  10. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  11. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  12. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  13. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  14. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  15. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  16. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  17. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  18. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  19. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  20. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  21. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  22. 2025A Convergence Analysis of Adaptive Optimizers under Floating-point Quantizationcited by 8
  23. 2019Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learningcited by 6
  24. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-trainingcited by 6
  25. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  26. 2026Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tailscited by 4
  27. 2025Stacey: Promoting Stochastic Steepest Descent via Accelerated ℓp-Smooth Nonconvex Optimizationcited by 3
  28. 2016On the Powerball Method for Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformers
  2. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics
  3. 2024Toward generalizable learning of all (linear) first-order methods via memory augmented Transformers
  4. 2026Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers
  5. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization
  6. 2026FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
  7. 2026Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
  8. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression