# TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning

arXiv: https://arxiv.org/abs/2007.11622 · HTML: https://mlsanity.com/paper/2007.11622/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 30 of 32)

- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning](https://mlsanity.com/paper/2402.15751/) (2024)
- [Federated Learning for Internet of Things: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2111.07494/) (2021)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training](https://mlsanity.com/paper/2410.19313/) (2024)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [Intelligent Model Update Strategy for Sequential Recommendation](https://mlsanity.com/paper/2302.07335/) (2023)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Federated Few-Shot Learning for Mobile NLP](https://mlsanity.com/paper/2212.05974/) (2022)
- [TinyML: Tools, Applications, Challenges, and Future Research Directions](https://mlsanity.com/paper/2303.13569/) (2023)
- [Efficient On-device Training via Gradient Filtering](https://mlsanity.com/paper/2301.00330/) (2023)
- [PoSynDA: Multi-Hypothesis Pose Synthesis Domain Adaptation for Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.09678/) (2023)
- [ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection](https://mlsanity.com/paper/2312.14227/) (2023)
- [Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers](https://mlsanity.com/paper/2305.19167/) (2023)
- [Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraint](https://mlsanity.com/paper/2312.05282/) (2023)
- [TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning](https://mlsanity.com/paper/2308.09201/) (2023)
- [Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2411.01179/) (2024)
- [Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation](https://mlsanity.com/paper/2409.07109/) (2024)
- [Prompt-Tuning SAM: From Generalist to Specialist with only 2048 Parameters and 16 Training Images](https://mlsanity.com/paper/2504.16739/) (2025)
- [Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning](https://mlsanity.com/paper/2505.14125/) (2025)
- [Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning](https://mlsanity.com/paper/2601.09076/) (2026)
- [Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures](https://mlsanity.com/paper/2303.16100/) (2023)
- [EffiPerception: an Efficient Framework for Various Perception Tasks](https://mlsanity.com/paper/2403.12317/) (2024)
- [TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge](https://mlsanity.com/paper/2603.09511/) (2026)
- [Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer Interaction: A Comprehensive Survey](https://mlsanity.com/paper/2401.06000/) (2024)
- [DeepMachining: Online Prediction of Machining Errors of Lathe Machines](https://mlsanity.com/paper/2403.16451/) (2024)
- [Distributed Convolutional Neural Network Training on Mobile and Edge Clusters](https://mlsanity.com/paper/2409.09083/) (2024)
- [Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2409.16278/) (2024)
- [Unsupervised Parameter Efficient Source-free Post-pretraining](https://mlsanity.com/paper/2502.21313/) (2025)
- [Dynamic Gradient Sparse Update for Edge Training](https://mlsanity.com/paper/2503.17959/) (2025)

## Abstract (arXiv)

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.