# Continual Named Entity Recognition without Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2310.14541 · HTML: https://mlsanity.com/paper/2310.14541/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)
- [ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2006.08658/) (2020)
- [RBC: Rectifying the Biased Context in Continual Semantic Segmentation](https://mlsanity.com/paper/2203.08404/) (2022)
- [Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization](https://mlsanity.com/paper/2203.13167/) (2022)
- [Do Not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting](https://mlsanity.com/paper/2102.01906/) (2021)
- [Learning "O" Helps for Learning More: Handling the Concealed Entity Problem for Class-incremental NER](https://mlsanity.com/paper/2210.04676/) (2022)

## Built on by (most cited first, 11 of 11)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?](https://mlsanity.com/paper/2410.17594/) (2024)
- [Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models](https://mlsanity.com/paper/2312.07887/) (2023)
- [Incremental Sequence Labeling: A Tale of Two Shifts](https://mlsanity.com/paper/2402.10447/) (2024)
- [Unveiling and Addressing Pseudo Forgetting in Large Language Models](https://mlsanity.com/paper/2411.11932/) (2024)
- [Federated Incremental Named Entity Recognition](https://mlsanity.com/paper/2411.11623/) (2024)
- [Can LLMs Learn New Concepts Incrementally without Forgetting?](https://mlsanity.com/paper/2402.08526/) (2024)
- [Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition](https://mlsanity.com/paper/2508.03259/) (2025)
- [Information-Theoretic Complementary Prompts for Improved Continual Text Classification](https://mlsanity.com/paper/2505.20933/) (2025)
- [GenCNER: A Generative Framework for Continual Named Entity Recognition](https://mlsanity.com/paper/2510.11444/) (2025)
- [Type-Balanced Contextual Learning for Incremental Named Entity Recognition](https://mlsanity.com/paper/2608.31038/) (2026)

## Abstract (arXiv)

Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely afflicted by catastrophic forgetting. This issue is intensified in CNER due to the consolidation of old entity types from previous steps into the non-entity type at each step, leading to what is known as the semantic shift problem of the non-entity type. In this paper, we introduce a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones, thereby more effectively mitigating the problem of catastrophic forgetting. Additionally, we develop a confidence-based pseudo-labeling for the non-entity type, i.e., predicting entity types using the old model to handle the semantic shift of the non-entity type. Following the pseudo-labeling process, we suggest an adaptive re-weighting type-balanced learning strategy to handle the issue of biased type distribution. We carried out comprehensive experiments on ten CNER settings using three different datasets. The results illustrate that our method significantly outperforms prior state-of-the-art approaches, registering an average improvement of 6.3% and 8.0% in Micro and Macro F1 scores, respectively.