# Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches

arXiv: https://arxiv.org/abs/1907.03799 · HTML: https://mlsanity.com/paper/1907.03799/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 30 of 46)

- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022)
- [Continual Normalization: Rethinking Batch Normalization for Online Continual Learning](https://mlsanity.com/paper/2203.16102/) (2022)
- [Class-Incremental Learning with Generative Classifiers](https://mlsanity.com/paper/2104.10093/) (2021)
- [A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays](https://mlsanity.com/paper/2110.10486/) (2021)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition](https://mlsanity.com/paper/2104.03841/) (2021)
- [Is Class-Incremental Enough for Continual Learning?](https://mlsanity.com/paper/2112.02925/) (2021)
- [Continual Learning at the Edge: Real-Time Training on Smartphone Devices](https://mlsanity.com/paper/2105.13127/) (2021)
- [Continual Learning in Deep Networks: an Analysis of the Last Layer](https://mlsanity.com/paper/2106.01834/) (2021)
- [Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation](https://mlsanity.com/paper/2106.15287/) (2021)
- [Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt](https://mlsanity.com/paper/2403.09857/) (2024)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Continual Learning on the Edge with TensorFlow Lite](https://mlsanity.com/paper/2105.01946/) (2021)
- [Class-Incremental Learning with Repetition](https://mlsanity.com/paper/2301.11396/) (2023)
- [Alleviating Representational Shift for Continual Fine-tuning](https://mlsanity.com/paper/2204.10535/) (2022)
- [Lifelong Continual Learning for Anomaly Detection: New Challenges, Perspectives, and Insights](https://mlsanity.com/paper/2303.07557/) (2023)
- [From MNIST to ImageNet and Back: Benchmarking Continual Curriculum Learning](https://mlsanity.com/paper/2303.11076/) (2023)
- [Continual Adaptation of Vision Transformers for Federated Learning](https://mlsanity.com/paper/2306.09970/) (2023)
- [Continual learning of longitudinal health records](https://mlsanity.com/paper/2112.11944/) (2021)
- [Region-Based Optimization in Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2412.11551/) (2024)
- [Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments](https://mlsanity.com/paper/2110.11006/) (2021)
- [Diagnosing Batch Normalization in Class Incremental Learning](https://mlsanity.com/paper/2202.08025/) (2022)
- [Generative Negative Replay for Continual Learning](https://mlsanity.com/paper/2204.05842/) (2022)
- [Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions](https://mlsanity.com/paper/2303.14115/) (2023)
- [Enabling On-device Continual Learning with Binary Neural Networks](https://mlsanity.com/paper/2401.09916/) (2024)

## Abstract (arXiv)

Robotic vision is a field where continual learning can play a significant role. An embodied agent operating in a complex environment subject to frequent and unpredictable changes is required to learn and adapt continuously. In the context of object recognition, for example, a robot should be able to learn (without forgetting) objects of never before seen classes as well as improving its recognition capabilities as new instances of already known classes are discovered. Ideally, continual learning should be triggered by the availability of short videos of single objects and performed on-line on on-board hardware with fine-grained updates. In this paper, we introduce a novel continual learning protocol based on the CORe50 benchmark and propose two rehearsal-free continual learning techniques, CWR* and AR1*, that can learn effectively even in the challenging case of nearly 400 small non-i.i.d. incremental batches. In particular, our experiments show that AR1* can outperform other state-of-the-art rehearsal-free techniques by more than 15% accuracy in some cases, with a very light and constant computational and memory overhead across training batches.