# Incremental Sequence Labeling: A Tale of Two Shifts

arXiv: https://arxiv.org/abs/2402.10447 · HTML: https://mlsanity.com/paper/2402.10447/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference](https://mlsanity.com/paper/2306.10790/) (2023)
- [Continual Named Entity Recognition without Catastrophic Forgetting](https://mlsanity.com/paper/2310.14541/) (2023)
- [Attending to Characters in Neural Sequence Labeling Models](https://mlsanity.com/paper/1611.04361/) (2016)
- [Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction](https://mlsanity.com/paper/2305.16559/) (2023)
- [Learning "O" Helps for Learning More: Handling the Concealed Entity Problem for Class-incremental NER](https://mlsanity.com/paper/2210.04676/) (2022)

## Built on by (most cited first, 5 of 5)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Cross-domain Named Entity Recognition via Graph Matching](https://mlsanity.com/paper/2408.00981/) (2024)
- [Federated Incremental Named Entity Recognition](https://mlsanity.com/paper/2411.11623/) (2024)
- [Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition](https://mlsanity.com/paper/2508.03259/) (2025)
- [GenCNER: A Generative Framework for Continual Named Entity Recognition](https://mlsanity.com/paper/2510.11444/) (2025)

## Abstract (arXiv)

The incremental sequence labeling task involves continuously learning new classes over time while retaining knowledge of the previous ones. Our investigation identifies two significant semantic shifts: E2O (where the model mislabels an old entity as a non-entity) and O2E (where the model labels a non-entity or old entity as a new entity). Previous research has predominantly focused on addressing the E2O problem, neglecting the O2E issue. This negligence results in a model bias towards classifying new data samples as belonging to the new class during the learning process. To address these challenges, we propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3). Motivated by the identified semantic shifts (E2O and O2E), IS3 aims to mitigate catastrophic forgetting in models. As for the E2O problem, we use knowledge distillation to maintain the model's discriminative ability for old entities. Simultaneously, to tackle the O2E problem, we alleviate the model's bias towards new entities through debiased loss and optimization levels. Our experimental evaluation, conducted on three datasets with various incremental settings, demonstrates the superior performance of IS3 compared to the previous state-of-the-art method by a significant margin.The data, code, and scripts are publicly available at https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.