mlsanityData to 7 Oct 2026

On Surprising Effectiveness of Masking Updates in Adaptive Optimizers

arXiv 2602.15322 ↗First posted 17 Feb 2026cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming recent state-of-the-art optimizers. Our analysis reveals that the random masking induces a curvature-dependent geometric regularization that smooths the optimization trajectory. Motivated by this finding, we introduce Momentum-aligned gradient masking (Magma), which modulates the masked updates using momentum-gradient alignment. Extensive LLM pre-training experiments show that Magma is a simple drop-in replacement for adaptive optimizers with consistent gains and negligible computational overhead. Notably, for the 1B model size, Magma reduces perplexity by over 19% and 9% compared to Adam and Muon, respectively.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

55 of 69 references.

References · 55

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
38 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  6. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  9. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  11. 2017Three Factors Influencing Minima in SGDcited by 270
  12. 2015Coordinate Descent Algorithmscited by 218
  13. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  14. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  15. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  16. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  17. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  18. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  19. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  20. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  21. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  22. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  23. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  24. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  25. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  26. 2020A Framework for Evaluating Gradient Leakage Attacks in Federated Learningcited by 63
  27. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  28. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  29. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  30. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  31. 2025In Search of Adam's Secret Saucecited by 40
  32. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  33. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  34. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  35. 2022Implicit regularization of dropoutcited by 24
  36. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  37. 2018On the Implicit Bias of Dropoutcited by 22
  38. 2017Let's Make Block Coordinate Descent Converge Faster: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergencecited by 16

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Dion3: Full-Stack Orthogonal Updatescited by 3
  2. 2026Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimizationcited by 2
  3. 2026FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneitycited by 0
  4. 2026Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNetscited by 0
  5. 2026How Much Regularization Survives Averaging? Update Masking in Federated Learningcited by 0