mlsanityData to 7 Oct 2026

Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Network

arXiv 2001.08885 ↗First posted 24 Jan 2020eess.AS cs.LG cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

Training machine learning models on mobile devices has the potential of improving both privacy and accuracy of the models. However, one of the major obstacles to achieving this goal is the memory limitation of mobile devices. Reducing training memory enables models with high-dimensional weight matrices, like automatic speech recognition (ASR) models, to be trained on-device. In this paper, we propose approximating the gradient matrices of deep neural networks using a low-rank parameterization as an avenue to save training memory. The low-rank gradient approximation enables more advanced, memory-intensive optimization techniques to be run on device. Our experimental results show that we can reduce the training memory by about 33.0% for Adam optimization. It uses comparable memory to momentum optimization and achieves a 4.5% relative lower word error rate on an ASR personalization task.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 3; 2024: 4; 2025: 5; 2026: 2.

20 papers.

Sort:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  3. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  4. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  5. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  6. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  7. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  8. 2023Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learningcited by 10
  9. 2021Improving Differentially Private SGD via Randomly Sparsified Gradientscited by 8
  10. 2022p-Meta: Towards On-device Deep Model Adaptationcited by 8
  11. 2022Federated Optimization of Smooth Loss Functionscited by 6
  12. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
8 more
  1. 2025AdaPM: a Partial Momentum Algorithm for LLM Trainingcited by 3
  2. 2022Gradient Descent for Low-Rank Functionscited by 2
  3. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 2
  4. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  5. 2023MobileASR: A resource-aware on-device learning framework for user voice personalization applications on mobile phonescited by 0
  6. 2024Weights Shuffling for Improving DPSGD in Transformer-based Modelscited by 0
  7. 2025Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layerscited by 0
  8. 2026SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptationcited by 0