mlsanityData to 14 Oct 2026

ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

arXiv 2312.14227 ↗First posted 21 Dec 2023cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5x more training speedup in wall-clock time and reduces energy consumption by 2x-3x more compared to the existing schemes, without noticeable accuracy loss.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 49 of its 85 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

49 of 85 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  12. 2016Layer Normalizationcited by 5,669
37 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2018Group Normalizationcited by 1,524
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2015Automatic differentiation in machine learning: a surveycited by 855
  11. 2019Towards Federated Learning at Scale: System Designcited by 832
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  13. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  14. 2019Importance Estimation for Neural Network Pruningcited by 541
  15. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  16. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  17. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  18. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  19. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 398
  20. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  21. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  22. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  23. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  24. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  25. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  26. 2022On-Device Training Under 256KB Memorycited by 160
  27. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  28. 2019Split Learning for collaborative deep learning in healthcarecited by 73
  29. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 71
  30. 2019PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfigurationcited by 41
  31. 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
  32. 2018K for the Price of 1: Parameter-efficient Multi-task and Transfer Learningcited by 34
  33. 2020TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learningcited by 32
  34. 2022Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloadingcited by 29
  35. 2015When Face Recognition Meets with Deep Learning: an Evaluation of Convolutional Neural Networks for Face Recognitioncited by 21
  36. 2019AdaFilter: Adaptive Filter Fine-tuning for Deep Transfer Learningcited by 12
  37. 2018Continuously Constructive Deep Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 2.

12 papers.

  1. 2025Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selectioncited by 8
  2. 2025BoTTA: Benchmarking on-device Test Time Adaptationcited by 4
  3. 2024Elastic On-Device LLM Servicecited by 3
  4. 2025DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Trainingcited by 3
  5. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3
  6. 2025Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Trainingcited by 2
  7. 2025A Two-Stage Data Selection Framework for Data-Efficient Model Training on Edge Devicescited by 2
  8. 2024AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environmentscited by 1
  9. 2025LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devicescited by 1
  10. 2025AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networkscited by 1
  11. 2026PowerLens: Taming LLM Agents for Safe and Personalized Mobile Power Managementcited by 0
  12. 2026FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUscited by 0