# SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning

arXiv: https://arxiv.org/abs/2511.22367 · HTML: https://mlsanity.com/paper/2511.22367/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5](https://mlsanity.com/paper/2110.07298/) (2021)
- [Human-inspired Episodic Memory for Infinite Context LLMs](https://mlsanity.com/paper/2407.09450/) (2024)
- [Investigating Continual Pretraining in Large Language Models: Insights and Implications](https://mlsanity.com/paper/2402.17400/) (2024)
- [Large language models can segment narrative events similarly to humans](https://mlsanity.com/paper/2301.10297/) (2023)
- [DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2502.11482/) (2025)
- [Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models](https://mlsanity.com/paper/2509.06100/) (2025)
- [How Relevant is Selective Memory Population in Lifelong Language Learning?](https://mlsanity.com/paper/2210.00940/) (2022)
- [AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning](https://mlsanity.com/paper/2509.17348/) (2025)

## Built on by (most cited first, 4 of 4)

- [On-Policy Replay for Continual Supervised Fine-Tuning](https://mlsanity.com/paper/2605.29495/) (2026)
- [Do Self-Evolving Agents Forget? Capability Degradation and Preservation in Lifelong LLM Agent Adaptation](https://mlsanity.com/paper/2605.09315/) (2026)
- [Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation](https://mlsanity.com/paper/2603.04688/) (2026)
- [LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agents](https://mlsanity.com/paper/2609.14138/) (2026)

## Abstract (arXiv)

Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.