# Adaptive Memory Replay for Continual Learning

arXiv: https://arxiv.org/abs/2404.12526 · HTML: https://mlsanity.com/paper/2404.12526/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 60 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)

## Built on by (most cited first, 22 of 22)

- [Prototype-Based Continual Learning with Label-free Replay Buffer and Cluster Preservation Loss](https://mlsanity.com/paper/2504.07240/) (2025)
- [RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection](https://mlsanity.com/paper/2508.13878/) (2025)
- [Same accuracy, twice as fast: continuous training surpasses retraining from scratch](https://mlsanity.com/paper/2502.21147/) (2025)
- [Few-Shot Vision-Language Action-Incremental Policy Learning](https://mlsanity.com/paper/2504.15517/) (2025)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)
- [Elastic Weight Consolidation Done Right for Continual Learning](https://mlsanity.com/paper/2603.18596/) (2026)
- [On Memory: A comparison of memory mechanisms in world models](https://mlsanity.com/paper/2512.06983/) (2025)
- [PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models](https://mlsanity.com/paper/2606.03598/) (2026)
- [FedTeddi: Temporal Drift and Divergence Aware Scheduling for Timely Federated Edge Learning](https://mlsanity.com/paper/2509.07342/) (2025)
- [Continual-learning for Modelling Low-Resource Languages from Large Language Models](https://mlsanity.com/paper/2601.05874/) (2026)
- [Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2602.20796/) (2026)
- [Reframing Long-Tailed Learning via Loss Landscape Geometry](https://mlsanity.com/paper/2603.21217/) (2026)
- [AIFIND: Artifact-Aware Interpreting Fine-Grained Alignment for Incremental Face Forgery Detection](https://mlsanity.com/paper/2604.16207/) (2026)
- [Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models](https://mlsanity.com/paper/2605.14938/) (2026)
- [ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing](https://mlsanity.com/paper/2605.14948/) (2026)
- [Rethinking Transfer in Continual Learning: A Replay-Based Realisation](https://mlsanity.com/paper/2607.15587/) (2026)
- [Replay to Remember: Retaining Domain Knowledge in Streaming Language Models](https://mlsanity.com/paper/2504.17780/) (2025)
- [Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offs](https://mlsanity.com/paper/2603.14271/) (2026)
- [Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training](https://mlsanity.com/paper/2603.17198/) (2026)
- [Challenges of Explainability in Continual Learning for Time Series Forecasting](https://mlsanity.com/paper/2607.19382/) (2026)
- [Adaptive Intrusion Detection System using Transformer-Based Neural Networks and Continual Learning Approach with Adversarial Investigation](https://mlsanity.com/paper/2608.04602/) (2026)
- [Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining](https://mlsanity.com/paper/2609.40089/) (2026)

## Abstract (arXiv)

Foundation Models (FMs) have become the hallmark of modern AI, however, these models are trained on massive data, leading to financially expensive training. Updating FMs as new data becomes available is important, however, can lead to `catastrophic forgetting', where models underperform on tasks related to data sub-populations observed too long ago. This continual learning (CL) phenomenon has been extensively studied, but primarily in a setting where only a small amount of past data can be stored. We advocate for the paradigm where memory is abundant, allowing us to keep all previous data, but computational resources are limited. In this setting, traditional replay-based CL approaches are outperformed by a simple baseline which replays past data selected uniformly at random, indicating that this setting necessitates a new approach. We address this by introducing a framework of adaptive memory replay for continual learning, where sampling of past data is phrased as a multi-armed bandit problem. We utilize Bolzmann sampling to derive a method which dynamically selects past data for training conditioned on the current task, assuming full data access and emphasizing training efficiency. Through extensive evaluations on both vision and language pre-training tasks, we demonstrate the effectiveness of our approach, which maintains high performance while reducing forgetting by up to 10% at no training efficiency cost.