# Latent Replay for Real-Time Continual Learning

arXiv: https://arxiv.org/abs/1912.01100 · HTML: https://mlsanity.com/paper/1912.01100/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2019.

## Built on by (most cited first, 30 of 97)

- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Constrained Few-shot Class-incremental Learning](https://mlsanity.com/paper/2203.16588/) (2022)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials](https://mlsanity.com/paper/2205.05467/) (2022)
- [PIVOT: Prompting for Video Continual Learning](https://mlsanity.com/paper/2212.04842/) (2022)
- [A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays](https://mlsanity.com/paper/2110.10486/) (2021)
- [Continual Learning for Blind Image Quality Assessment](https://mlsanity.com/paper/2102.09717/) (2021)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [Continual Learning with Low Rank Adaptation](https://mlsanity.com/paper/2311.17601/) (2023)
- [Continual Object Detection: A review of definitions, strategies, and challenges](https://mlsanity.com/paper/2205.15445/) (2022)
- [BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning](https://mlsanity.com/paper/2305.04769/) (2023)
- [Condensed Composite Memory Continual Learning](https://mlsanity.com/paper/2102.09890/) (2021)
- [Continual Learning at the Edge: Real-Time Training on Smartphone Devices](https://mlsanity.com/paper/2105.13127/) (2021)
- [The Future of Continual Learning in the Era of Foundation Models: Three Key Directions](https://mlsanity.com/paper/2506.03320/) (2025)
- [HFedMS: Heterogeneous Federated Learning with Memorable Data Semantics in Industrial Metaverse](https://mlsanity.com/paper/2211.03300/) (2022)
- [GRASP: A Rehearsal Policy for Efficient Online Continual Learning](https://mlsanity.com/paper/2308.13646/) (2023)
- [Intelligence at the Extreme Edge: A Survey on Reformable TinyML](https://mlsanity.com/paper/2204.00827/) (2022)
- [Overcoming the Stability Gap in Continual Learning](https://mlsanity.com/paper/2306.01904/) (2023)
- [Unveiling the Anomalies in an Ever-Changing World: A Benchmark for Pixel-Level Anomaly Detection in Continual Learning](https://mlsanity.com/paper/2403.15463/) (2024)
- [Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings](https://mlsanity.com/paper/2103.02886/) (2021)
- [Learn the Time to Learn: Replay Scheduling in Continual Learning](https://mlsanity.com/paper/2209.08660/) (2022)
- [Shoggoth: Towards Efficient Edge-Cloud Collaborative Real-Time Video Inference via Adaptive Online Learning](https://mlsanity.com/paper/2306.15333/) (2023)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)

## Abstract (arXiv)

Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art performance on complex video benchmarks such as CORe50 NICv2 (with nearly 400 small and highly non-i.i.d. batches) and OpenLORIS. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the deployment of the proposed technique on a smartphone device.