mlsanityData to 14 Oct 2026

TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge

arXiv 2307.09988 ↗First posted 19 Jul 2023cs.LG cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

On-device training is essential for user personalisation and privacy. With the pervasiveness of IoT devices and microcontroller units (MCUs), this task becomes more challenging due to the constrained memory and compute resources, and the limited availability of labelled user data. Nonetheless, prior works neglect the data scarcity issue, require excessively long training time (e.g. a few hours), or induce substantial accuracy loss (>10%). In this paper, we propose TinyTrain, an on-device training approach that drastically reduces training time by selectively updating parts of the model and explicitly coping with data scarcity. TinyTrain introduces a task-adaptive sparse-update method that dynamically selects the layer/channel to update based on a multi-objective criterion that jointly captures user data, the memory, and the compute capabilities of the target device, leading to high accuracy on unseen tasks with reduced computation and memory footprint. TinyTrain outperforms vanilla fine-tuning of the entire network by 3.6-5.0% in accuracy, while reducing the backward-pass memory and computation cost by up to 1,098x and 7.68x, respectively. Targeting broadly used real-world edge devices, TinyTrain achieves 9.5x faster and 3.5x more energy-efficient training over status-quo approaches, and 2.23x smaller memory footprint than SOTA methods, while remaining within the 1 MB memory envelope of MCU-grade platforms.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 46 of its 77 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

46 of 77 references.

References · 46

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2016Matching Networks for One Shot Learningcited by 2,255
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  11. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
34 more
  1. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  3. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  4. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  6. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  7. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  8. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 315
  9. 2018How to train your MAMLcited by 309
  10. 2019A Baseline for Few-Shot Image Classificationcited by 298
  11. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  12. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  13. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  14. 2020TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systemscited by 165
  15. 2022On-Device Training Under 256KB Memorycited by 160
  16. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
  17. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 94
  18. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  19. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
  20. 2021Cross-domain Few-shot Learning with Task-specific Adapterscited by 80
  21. 2021TinyOL: TinyML with Online-Learning on Microcontrollerscited by 65
  22. 2018SqueezeNext: Hardware-Aware Neural Network Designcited by 61
  23. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  24. 2019Low-Memory Neural Network Training: A Technical Reportcited by 57
  25. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  26. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  27. 2021Shallow Bayesian Meta Learning for Real-World Few-Shot Recognitioncited by 48
  28. 2020Dynamic Tensor Rematerializationcited by 46
  29. 2019E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savingscited by 36
  30. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  31. 2022Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloadingcited by 29
  32. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26
  33. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16
  34. 2022p-Meta: Towards On-device Deep Model Adaptationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Modelscited by 8
  2. 2024Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensorscited by 7
  3. 2025Benchmarking Ultra-Low-Power μNPUscited by 6
  4. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  5. 2025Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Trainingcited by 2
  6. 2025DecoHD: Decomposed Hyperdimensional Classification under Extreme Memory Budgetscited by 2
  7. 2025LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devicescited by 1
  8. 2025Onboard Optimization and Learning: A Surveycited by 1
  9. 2025AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networkscited by 1
  10. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  11. 2025MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devicescited by 0
  12. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0
6 more
  1. 2025FTTE: Enabling Federated and Resource-Constrained Deep Edge Intelligencecited by 0
  2. 2025Memory Constrained Dynamic Subnetwork Update for Transfer Learningcited by 0
  3. 2025Multi-modal On-Device Learning for Monocular Depth Estimation on Ultra-low-power MCUscited by 0
  4. 2026DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devicescited by 0
  5. 2026What changes after deployment? A survey on On-device Learning in TinyMLcited by 0
  6. 2026Versatile On-device Adaptation at the Edge by Unifying Few-shot, Zero-shot, Continual, and In-context Learningcited by 0