# Continual Learning via Self-Probe Gradients

arXiv: https://arxiv.org/abs/2609.32771 · HTML: https://mlsanity.com/paper/2609.32771/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal](https://mlsanity.com/paper/2403.01244/) (2024)
- [Optimal Continual Learning has Perfect Memory and is NP-hard](https://mlsanity.com/paper/2006.05188/) (2020)
- [CPR: Classifier-Projection Regularization for Continual Learning](https://mlsanity.com/paper/2006.07326/) (2020)
- [Computationally Budgeted Continual Learning: What Does Matter?](https://mlsanity.com/paper/2303.11165/) (2023)
- [Continual evaluation for lifelong learning: Identifying the stability gap](https://mlsanity.com/paper/2205.13452/) (2022)
- [Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection](https://mlsanity.com/paper/2502.06042/) (2025)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [How Efficient Are Today's Continual Learning Algorithms?](https://mlsanity.com/paper/2303.18171/) (2023)
- [Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition](https://mlsanity.com/paper/2410.01380/) (2024)
- [Replaying pre-training data improves fine-tuning](https://mlsanity.com/paper/2603.04964/) (2026)
- [Hyperparameters in Continual Learning: A Reality Check](https://mlsanity.com/paper/2403.09066/) (2024)
- [Forget Forgetting: Continual Learning in a World of Abundant Memory](https://mlsanity.com/paper/2502.07274/) (2025)

## Built on by (most cited first, 1 of 1)

- [Self-Preservation in Language Models](https://mlsanity.com/paper/2610.12602/) (2026)

## Abstract (arXiv)

Adapting pretrained models to new data can cause catastrophic forgetting of previously learned behavior. When only a few past samples remain, they give continual learning methods sparse and narrow evidence about what to preserve. We show that language models can expand this evidence through self-probing, in which the frozen model generates new inputs from the retained samples and records its own predictions on them. Unlike prior work that replays such data as training examples, our method, CPLUS uses self-probe and past-sample gradients to scale down parameter updates that conflict with prior behavior. Experiments with five language models on four benchmarks show three results. First, the same probes preserve more prior behavior as gradient signals than as replay data. Second, CPLUS learns the new data while consistently reducing forgetting more than existing baselines, especially when past data are scarce, and this protection extends to benchmarks not used for training. Third, we observe that CPLUS also becomes more effective as models grow: within the Qwen3 model family, it recovers an increasing share of the forgetting caused by standard fine-tuning.