# ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection

arXiv: https://arxiv.org/abs/2312.14227 · HTML: https://mlsanity.com/paper/2312.14227/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 85 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)

## Built on by (most cited first, 12 of 12)

- [Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection](https://mlsanity.com/paper/2504.13938/) (2025)
- [BoTTA: Benchmarking on-device Test Time Adaptation](https://mlsanity.com/paper/2504.10149/) (2025)
- [Elastic On-Device LLM Service](https://mlsanity.com/paper/2409.09071/) (2024)
- [DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Training](https://mlsanity.com/paper/2507.07149/) (2025)
- [FedEL: Federated Elastic Learning for Heterogeneous Devices](https://mlsanity.com/paper/2509.16902/) (2025)
- [Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Training](https://mlsanity.com/paper/2504.09405/) (2025)
- [A Two-Stage Data Selection Framework for Data-Efficient Model Training on Edge Devices](https://mlsanity.com/paper/2505.16563/) (2025)
- [AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments](https://mlsanity.com/paper/2410.08256/) (2024)
- [LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devices](https://mlsanity.com/paper/2503.15889/) (2025)
- [AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/2510.03101/) (2025)
- [PowerLens: Taming LLM Agents for Safe and Personalized Mobile Power Management](https://mlsanity.com/paper/2603.19584/) (2026)
- [FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs](https://mlsanity.com/paper/2607.21624/) (2026)

## Abstract (arXiv)

On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5x more training speedup in wall-clock time and reduces energy consumption by 2x-3x more compared to the existing schemes, without noticeable accuracy loss.