mlsanityData to 13 Oct 2026

Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning

arXiv 2505.05086 ↗First posted 8 May 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to 120.09× compared to vanilla training, while also reducing overall training FLOPs up to 1.86× when evaluated on traditional benchmarks.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

26 of 38 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,849
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,786
  5. 2016Pyramid Scene Parsing Networkcited by 2,474
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  7. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,159
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,268
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 611
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 565
  12. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
14 more
  1. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 263
  2. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 237
  3. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 231
  4. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  5. 2020The Computational Limits of Deep Learningcited by 190
  6. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 162
  7. 2022On-Device Training Under 256KB Memorycited by 160
  8. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  9. 2020Deep Learning for Financial Applications : A Surveycited by 91
  10. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  11. 2023Efficient On-device Training via Gradient Filteringcited by 13
  12. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  13. 2023Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraintcited by 6
  14. 2024Activation Map Compression through Tensor Decomposition for Deep Learningcited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learningcited by 3
  2. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  3. 2026CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecastingcited by 1
  4. 2025Efficient Resource-Constrained Training of Transformers via Subspace Optimizationcited by 0
  5. 2025Memory Constrained Dynamic Subnetwork Update for Transfer Learningcited by 0