# A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays

arXiv: https://arxiv.org/abs/2110.10486 · HTML: https://mlsanity.com/paper/2110.10486/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems](https://mlsanity.com/paper/2010.08678/) (2020)
- [Memory Efficient Experience Replay for Streaming Learning](https://mlsanity.com/paper/1809.05922/) (2018)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [Latent Replay for Real-Time Continual Learning](https://mlsanity.com/paper/1912.01100/) (2019)
- [Benchmarking TinyML Systems: Challenges and Direction](https://mlsanity.com/paper/2003.04821/) (2020)
- [TinyOL: TinyML with Online-Learning on Microcontrollers](https://mlsanity.com/paper/2103.08295/) (2021)
- [On-Device Machine Learning: An Algorithms and Learning Theory Perspective](https://mlsanity.com/paper/1911.00623/) (2019)
- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://mlsanity.com/paper/1802.04799/) (2018)
- [DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs](https://mlsanity.com/paper/2008.07127/) (2020)
- [Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches](https://mlsanity.com/paper/1907.03799/) (2019)
- [Spiking Neural Networks and Online Learning: An Overview and Perspectives](https://mlsanity.com/paper/1908.08019/) (2019)
- [Robust High-dimensional Memory-augmented Neural Networks](https://mlsanity.com/paper/2010.01939/) (2020)
- [CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions](https://mlsanity.com/paper/2009.09929/) (2020)
- [Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles](https://mlsanity.com/paper/2007.00302/) (2020)
- [Batch-level Experience Replay with Review for Continual Learning](https://mlsanity.com/paper/2007.05683/) (2020)
- [Quantization-Guided Training for Compact TinyML Models](https://mlsanity.com/paper/2103.06231/) (2021)
- [Memory-Latency-Accuracy Trade-offs for Continual Learning on a RISC-V Extreme-Edge Node](https://mlsanity.com/paper/2007.13631/) (2020)

## Built on by (most cited first, 30 of 37)

- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Constrained Few-shot Class-incremental Learning](https://mlsanity.com/paper/2203.16588/) (2022)
- [Divide and not forget: Ensemble of selectively trained experts in Continual Learning](https://mlsanity.com/paper/2401.10191/) (2024)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)
- [Intelligence at the Extreme Edge: A Survey on Reformable TinyML](https://mlsanity.com/paper/2204.00827/) (2022)
- [Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers](https://mlsanity.com/paper/2305.19167/) (2023)
- [Online Continual Learning for Robust Indoor Object Recognition](https://mlsanity.com/paper/2307.09827/) (2023)
- [On TinyML and Cybersecurity: Electric Vehicle Charging Infrastructure Use Case](https://mlsanity.com/paper/2404.16894/) (2024)
- [QCore: Data-Efficient, On-Device Continual Calibration for Quantized Models -- Extended Version](https://mlsanity.com/paper/2404.13990/) (2024)
- [Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learning](https://mlsanity.com/paper/2410.18378/) (2024)
- [FeTrIL++: Feature Translation for Exemplar-Free Class-Incremental Learning with Hill-Climbing](https://mlsanity.com/paper/2403.07406/) (2024)
- [TinySV: Speaker Verification in TinyML with On-device Learning](https://mlsanity.com/paper/2406.01655/) (2024)
- [RedMule: A Mixed-Precision Matrix-Matrix Operation Engine for Flexible and Energy-Efficient On-Chip Linear Algebra and TinyML Training Acceleration](https://mlsanity.com/paper/2301.03904/) (2023)
- [TinyReptile: TinyML with Federated Meta-Learning](https://mlsanity.com/paper/2304.05201/) (2023)
- [Enabling On-device Continual Learning with Binary Neural Networks](https://mlsanity.com/paper/2401.09916/) (2024)
- [Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond](https://mlsanity.com/paper/2408.02983/) (2024)
- [Latent Distillation for Continual Object Detection at the Edge](https://mlsanity.com/paper/2409.01872/) (2024)
- [Hadamard Domain Training with Integers for Class Incremental Quantized Learning](https://mlsanity.com/paper/2310.03675/) (2023)
- [12 mJ per Class On-Device Online Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2403.07851/) (2024)
- [On-device Learning of EEGNet-based Network For Wearable Motor Imagery Brain-Computer Interface](https://mlsanity.com/paper/2409.00083/) (2024)
- [MicroFlow: An Efficient Rust-Based Inference Engine for TinyML](https://mlsanity.com/paper/2409.19432/) (2024)
- [Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data](https://mlsanity.com/paper/2505.24852/) (2025)
- [TActiLE: Tiny Active LEarning for wearable devices](https://mlsanity.com/paper/2505.01160/) (2025)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [RockNet: Distributed Learning on Ultra-Low-Power Devices](https://mlsanity.com/paper/2510.13320/) (2025)
- [A Conditioned Unsupervised Regression Framework Attuned to the Dynamic Nature of Data Streams](https://mlsanity.com/paper/2312.07682/) (2023)
- [Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks](https://mlsanity.com/paper/2503.07107/) (2025)
- [Generative Binary Memory: Pseudo-Replay Class-Incremental Learning on Binarized Embeddings](https://mlsanity.com/paper/2503.10333/) (2025)
- [Bridging Generalization and Personalization in Human Activity Recognition via On-Device Few-Shot Learning](https://mlsanity.com/paper/2508.15413/) (2025)

## Abstract (arXiv)

In the last few years, research and development on Deep Learning models and techniques for ultra-low-power devices in a word, TinyML has mainly focused on a train-then-deploy assumption, with static models that cannot be adapted to newly collected data without cloud-based data collection and fine-tuning. Latent Replay-based Continual Learning (CL) techniques[1] enable online, serverless adaptation in principle, but so farthey have still been too computation and memory-hungry for ultra-low-power TinyML devices, which are typically based on microcontrollers. In this work, we introduce a HW/SW platform for end-to-end CL based on a 10-core FP32-enabled parallel ultra-low-power (PULP) processor. We rethink the baseline Latent Replay CL algorithm, leveraging quantization of the frozen stage of the model and Latent Replays (LRs) to reduce their memory cost with minimal impact on accuracy. In particular, 8-bit compression of the LR memory proves to be almost lossless (-0.26% with 3000LR) compared to the full-precision baseline implementation, but requires 4x less memory, while 7-bit can also be used with an additional minimal accuracy degradation (up to 5%). We also introduce optimized primitives for forward and backward propagation on the PULP processor. Our results show that by combining these techniques, continual learning can be achieved in practice using less than 64MB of memory an amount compatible with embedding in TinyML devices. On an advanced 22nm prototype of our platform, called VEGA, the proposed solution performs onaverage 65x faster than a low-power STM32 L4 microcontroller, being 37x more energy efficient enough for a lifetime of 535h when learning a new mini-batch of data once every minute.