mlsanityData to 14 Oct 2026

GradPower: Powering Gradients for Faster Language Model Pre-Training

arXiv 2505.24275 ↗First posted 30 May 2025cs.LG math.OC stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector g=(gi)i, GradPower first applies the elementwise sign-power transformation: varphip(g)=(rm sign(gi)|gi|p)i for a fixed p>0, and then feeds the transformed gradient into a base optimizer. Notably, GradPower requires only a single-line code change and no modifications to the base optimizer's internal logic, including the hyperparameters. When applied to Adam (termed AdamPower), GradPower consistently achieves lower terminal loss across diverse architectures (LLaMA, Qwen2MoE), parameter scales (66M to 2B), datasets (C4, OpenWebText), and learning-rate schedules (cosine, warmup-stable-decay). The most pronounced gains are observed when training modern mixture-of-experts models with warmup-stable-decay schedules. GradPower also integrates seamlessly with other state-of-the-art optimizers, such as Muon, yielding further improvements. Finally, we provide theoretical analyses that reveal the underlying mechanism of GradPower and highlight the influence of gradient noise.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

57 of 74 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2021Scaling Vision Transformerscited by 999
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
  13. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  14. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  15. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  16. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  17. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  18. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  19. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  20. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  21. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  22. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  23. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  24. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  25. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  26. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  27. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  28. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  29. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  30. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  31. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  32. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  33. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  34. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  35. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  36. 2025In Search of Adam's Secret Saucecited by 40
  37. 2024Does SGD really happen in tiny subspaces?cited by 36
  38. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  2. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  3. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 4
  4. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  5. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 3
  6. 2026Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratiocited by 0
  7. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  8. 2026CAST: History-Adaptive Spectral Clipping for LLM Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
  2. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
  3. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs
  4. 2024Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models
  5. 2026Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
  6. 2025Pre-training under infinite compute
  7. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
  8. 2025Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation