Breaking On-device Training Memory Wall: A Systematic Survey
Not on the map. Why?
Abstract (arXiv)
On-device training has become an increasingly popular approach to machine learning, enabling models to be trained directly on mobile and edge devices. However, a major challenge in this area is the limited memory available on these devices, which can severely restrict the size and complexity of the models that can be trained. In this systematic survey, we aim to explore the current state-of-the-art techniques for breaking on-device training memory walls, focusing on methods that can enable larger and more complex models to be trained on resource-constrained devices. Specifically, we first analyze the key factors that contribute to the phenomenon of memory walls encountered during on-device training. Then, we present a comprehensive literature review of on-device training, which addresses the issue of memory limitations. Finally, we summarize on-device training and highlight the open problems for future research. By providing a comprehensive overview of these techniques and their effectiveness in breaking memory walls, we hope to help researchers and practitioners in this field navigate the rapidly evolving landscape of on-device training.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 50 references that are in the map. First posted on arXiv 17 Jun 2023.
Builds on
23 of 50 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2017Mixed Precision Trainingcited by 1,156
- 2016Training Deep Nets with Sublinear Memory Costcited by 848
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
- 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
- 2021A Universal Law of Robustness via Isoperimetrycited by 151
- 2020Sparse GPU Kernels for Deep Learningcited by 141
11 more
- 2018NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Visioncited by 68
- 2019Low-Memory Neural Network Training: A Technical Reportcited by 57
- 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
- 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 35
- 2022Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloadingcited by 29
- 2022POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Pagingcited by 26
- 2022Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learningcited by 23
- 2018Dynamic Sparse Graph for Efficient Deep Learningcited by 18
- 2022OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networkscited by 6
- 2023Memory-adaptive Depth-wise Heterogeneous Federated Learningcited by 6
- 2022Swan: A Neural Engine for Efficient DNN Training on Smartphone SoCscited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.