mlsanityData to 14 Oct 2026

On-Device Training Under 256KB Memory

arXiv 2206.15472 ↗First posted 30 Jun 2022cs.CVcited by 160

Not on the map. Why?

Abstract (arXiv)

On-device training enables the model to adapt to new data collected from the sensors by fine-tuning a pre-trained model. Users can benefit from customized AI models without having to transfer the data to the cloud, protecting the privacy. However, the training memory consumption is prohibitive for IoT devices that have tiny memory resources. We propose an algorithm-system co-design framework to make on-device training possible with only 256KB of memory. On-device training faces two unique challenges: (1) the quantized graphs of neural networks are hard to optimize due to low bit-precision and the lack of normalization; (2) the limited hardware resource does not allow full back-propagation. To cope with the optimization difficulty, we propose Quantization-Aware Scaling to calibrate the gradient scales and stabilize 8-bit quantized training. To reduce the memory footprint, we propose Sparse Update to skip the gradient computation of less important layers and sub-tensors. The algorithm innovation is implemented by a lightweight training system, Tiny Training Engine, which prunes the backward computation graph to support sparse updates and offload the runtime auto-differentiation to compile time. Our framework is the first solution to enable tiny on-device training of convolutional neural networks under 256KB SRAM and 1MB Flash without auxiliary memory, using less than 1/1000 of the memory of PyTorch and TensorFlow while matching the accuracy on tinyML application VWW. Our study enables IoT devices not only to perform inference but also to continuously adapt to new data for on-device lifelong learning. A video demo can be found here: https://youtu.be/0pUFZYdoMY8.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

49 of 76 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
37 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  2. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  7. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  8. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  9. 2018Do Better ImageNet Models Transfer Better?cited by 565
  10. 2017Large Batch Training of Convolutional Networkscited by 535
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  13. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  14. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  15. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  16. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  17. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  18. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  19. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  20. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  21. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  22. 2018CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUscited by 121
  23. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 104
  24. 2020Benchmarking TinyML Systems: Challenges and Directioncited by 91
  25. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
  26. 2020MNN: A Universal and Efficient Inference Enginecited by 65
  27. 2021TinyOL: TinyML with Online-Learning on Microcontrollerscited by 65
  28. 2020DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUscited by 58
  29. 2020μNAS: Constrained Neural Architecture Search for Microcontrollerscited by 55
  30. 2019SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollerscited by 52
  31. 2019Visual Wake Words Datasetcited by 46
  32. 2018K for the Price of 1: Parameter-efficient Multi-task and Transfer Learningcited by 34
  33. 2019Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollerscited by 33
  34. 2020TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learningcited by 32
  35. 2020Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devicescited by 27
  36. 2019Neural networks on microcontrollers: saving memory at inference via operator reorderingcited by 15
  37. 2021TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNTcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 28; 2024: 55; 2025: 43; 2026: 30.

160 papers.

Sort:

By area:

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  2. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  3. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  4. 2024Tiny Machine Learning: Progress and Futurescited by 40
  5. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  6. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  9. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  10. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 23
  11. 2024Adapting to Distribution Shift by Visual Domain Prompt Generationcited by 23
  12. 2022On-device Training: A First Overview on Existing Systemscited by 19
38 more
  1. 2023AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systemscited by 19
  2. 2024Multisize Dataset Condensationcited by 19
  3. 2023TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edgecited by 18
  4. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  5. 2023Unleashing the Potential of Spiking Neural Networks by Dynamic Confidencecited by 15
  6. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  7. 2023Efficient On-device Training via Gradient Filteringcited by 13
  8. 2024On-Device Domain Learning for Keyword Spotting on Low-Power Extreme Edge Embedded Systemscited by 13
  9. 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
  10. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
  11. 2023Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagationcited by 12
  12. 2023ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selectioncited by 12
  13. 2023LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platformscited by 11
  14. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  15. 2023On-Device Unsupervised Image Segmentationcited by 10
  16. 2023Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllerscited by 10
  17. 2023To Spike or Not To Spike: A Digital Hardware Perspective on Deep Learning Accelerationcited by 10
  18. 2023Efficient Low-rank Backpropagation for Vision Transformer Adaptationcited by 10
  19. 2024Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generationcited by 9
  20. 2025A Survey on Wi-Fi Sensing Generalizability: Taxonomy, Techniques, Datasets, and Future Research Prospectscited by 9
  21. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  22. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  23. 2024Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensorscited by 7
  24. 2025SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsitycited by 7
  25. 2022OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networkscited by 6
  26. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  27. 2023Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraintcited by 6
  28. 2024Activation Map Compression through Tensor Decomposition for Deep Learningcited by 6
  29. 2025Benchmarking Ultra-Low-Power μNPUscited by 6
  30. 2023TinyAD: Memory-efficient anomaly detection for time series data in Industrial IoTcited by 5
  31. 2023Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Modelscited by 5
  32. 2024Lifelong Intelligence Beyond the Edge using Hyperdimensional Computingcited by 5
  33. 2024On-device Online Learning and Semantic Management of TinyML Systemscited by 5
  34. 2024TinySV: Speaker Verification in TinyML with On-device Learningcited by 5
  35. 2024On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollerscited by 5
  36. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  37. 2025Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learningcited by 5
  38. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 5

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
  2. 2023Cost-Driven Hardware-Software Co-Optimization of Machine Learning Pipelines
  3. 2023A 3 TOPS/W RISC-V Parallel Cluster for Inference of Fine-Grain Mixed-Precision Quantized Neural Networks
  4. 2021Quantization and Deployment of Deep Neural Networks on Microcontrollers
  5. 2023On-Device Learning with Binary Neural Networks
  6. 2022YONO: Modeling Multiple Heterogeneous Neural Networks on Microcontrollers
  7. 2021Enabling Binary Neural Network Training on the Edge
  8. 2021Pre-Quantized Deep Learning Models Codified in ONNX to Enable Hardware/Software Co-Design