# SIESTA: Efficient Online Continual Learning with Sleep

arXiv: https://arxiv.org/abs/2303.10725 · HTML: https://mlsanity.com/paper/2303.10725/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 110 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 30 of 66)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)

## Built on by (most cited first, 24 of 24)

- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [Summarizing Stream Data for Memory-Constrained Online Continual Learning](https://mlsanity.com/paper/2305.16645/) (2023)
- [GRASP: A Rehearsal Policy for Efficient Online Continual Learning](https://mlsanity.com/paper/2308.13646/) (2023)
- [Overcoming the Stability Gap in Continual Learning](https://mlsanity.com/paper/2306.01904/) (2023)
- [Addressing Concept Shift in Online Time Series Forecasting: Detect-then-Adapt](https://mlsanity.com/paper/2403.14949/) (2024)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning](https://mlsanity.com/paper/2502.10691/) (2025)
- [Wake-Sleep Consolidated Learning](https://mlsanity.com/paper/2401.08623/) (2023)
- [Scalable Strategies for Continual Learning with Replay](https://mlsanity.com/paper/2505.12512/) (2025)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [Low-redundancy Distillation for Continual Learning](https://mlsanity.com/paper/2309.16117/) (2023)
- [Hadamard Domain Training with Integers for Class Incremental Quantized Learning](https://mlsanity.com/paper/2310.03675/) (2023)
- [Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning](https://mlsanity.com/paper/2505.14125/) (2025)
- [Online Continual Learning via Spiking Neural Networks with Sleep Enhanced Latent Replay](https://mlsanity.com/paper/2507.02901/) (2025)
- [Distribution-aware Online Continual Learning for Urban Spatio-Temporal Forecasting](https://mlsanity.com/paper/2411.15893/) (2024)
- [A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization](https://mlsanity.com/paper/2503.06385/) (2025)
- [Efficient Continual Learning through Frequency Decomposition and Integration](https://mlsanity.com/paper/2503.22175/) (2025)
- [Understanding the Benefits of Image Augmentations](https://mlsanity.com/paper/2306.06254/) (2023)
- [TinyCL: An Efficient Hardware Architecture for Continual Learning on Autonomous Systems](https://mlsanity.com/paper/2402.09780/) (2024)
- [Brain-Inspired Continual Learning-Robust Feature Distillation and Re-Consolidation for Class Incremental Learning](https://mlsanity.com/paper/2404.14588/) (2024)
- [Patch-Based Contrastive Learning and Memory Consolidation for Online Unsupervised Continual Learning](https://mlsanity.com/paper/2409.16391/) (2024)
- [Dream2Learn: Structured Generative Dreaming for Continual Learning](https://mlsanity.com/paper/2603.01935/) (2026)
- [Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning](https://mlsanity.com/paper/2605.14785/) (2026)
- [SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition](https://mlsanity.com/paper/2606.00732/) (2026)

## Abstract (arXiv)

In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we propose a novel continual learning method, SIESTA based on wake/sleep framework for training, which is well aligned to the needs of on-device learning. The major goal of SIESTA is to advance compute efficient continual learning so that DNNs can be updated efficiently using far less time and energy. The principal innovations of SIESTA are: 1) rapid online updates using a rehearsal-free, backpropagation-free, and data-driven network update rule during its wake phase, and 2) expedited memory consolidation using a compute-restricted rehearsal policy during its sleep phase. For memory efficiency, SIESTA adapts latent rehearsal using memory indexing from REMIND. Compared to REMIND and prior arts, SIESTA is far more computationally efficient, enabling continual learning on ImageNet-1K in under 2 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications.