# ErrorEraser: Unlearning Data Bias for Improved Continual Learning

arXiv: https://arxiv.org/abs/2506.09347 · HTML: https://mlsanity.com/paper/2506.09347/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 32)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models](https://mlsanity.com/paper/2305.10120/) (2023)
- [Learning to Learn from Noisy Labeled Data](https://mlsanity.com/paper/1812.05214/) (2018)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Boundary Unlearning](https://mlsanity.com/paper/2303.11570/) (2023)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers](https://mlsanity.com/paper/2301.11578/) (2023)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [Dealing with Cross-Task Class Discrimination in Online Continual Learning](https://mlsanity.com/paper/2305.14657/) (2023)
- [Regularizing Second-Order Influences for Continual Learning](https://mlsanity.com/paper/2304.10177/) (2023)
- [Learning to Prompt Knowledge Transfer for Open-World Continual Learning](https://mlsanity.com/paper/2312.14990/) (2023)
- [Unlearning during Learning: An Efficient Federated Machine Unlearning Method](https://mlsanity.com/paper/2405.15474/) (2024)
- [CNLL: A Semi-supervised Approach For Continual Noisy Label Learning](https://mlsanity.com/paper/2204.09881/) (2022)
- [Task Relation-aware Continual User Representation Learning](https://mlsanity.com/paper/2306.01792/) (2023)
- [Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Grouping](https://mlsanity.com/paper/2502.20032/) (2025)

## Built on by (most cited first, 3 of 3)

- [The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?](https://mlsanity.com/paper/2509.22580/) (2025)
- [TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2607.21111/) (2026)
- [UFO: A Unified Flow-Oriented Framework for Robust Continual Graph Learning](https://mlsanity.com/paper/2605.09862/) (2026)

## Abstract (arXiv)

Continual Learning (CL) primarily aims to retain knowledge to prevent catastrophic forgetting and transfer knowledge to facilitate learning new tasks. Unlike traditional methods, we propose a novel perspective: CL not only needs to prevent forgetting, but also requires intentional forgetting.This arises from existing CL methods ignoring biases in real-world data, leading the model to learn spurious correlations that transfer and amplify across tasks. From feature extraction and prediction results, we find that data biases simultaneously reduce CL's ability to retain and transfer knowledge. To address this, we propose ErrorEraser, a universal plugin that removes erroneous memories caused by biases in CL, enhancing performance in both new and old tasks. ErrorEraser consists of two modules: Error Identification and Error Erasure. The former learns the probability density distribution of task data in the feature space without prior knowledge, enabling accurate identification of potentially biased samples. The latter ensures only erroneous knowledge is erased by shifting the decision space of representative outlier samples. Additionally, an incremental feature distribution learning strategy is designed to reduce the resource overhead during error identification in downstream tasks. Extensive experimental results show that ErrorEraser significantly mitigates the negative impact of data biases, achieving higher accuracy and lower forgetting rates across three types of CL methods. The code is available at https://github.com/diadai/ErrorEraser.