mlsanityData to 14 Oct 2026

Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization

arXiv 2512.13034 ↗First posted 15 Dec 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

23 of 32 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  12. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
11 more
  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  3. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  4. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  5. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  6. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  7. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  8. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  9. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  10. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  11. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 36

Built on by

Not cited yet.