# InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective

arXiv: https://arxiv.org/abs/2310.06362 · HTML: https://mlsanity.com/paper/2310.06362/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Continual Learning for Text Classification with Information Disentanglement Based Regularization](https://mlsanity.com/paper/2104.05489/) (2021)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Unbiased Supervised Contrastive Learning](https://mlsanity.com/paper/2211.05568/) (2022)
- [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.02714/) (2021)
- [Consistent Representation Learning for Continual Relation Extraction](https://mlsanity.com/paper/2203.02721/) (2022)
- [Improving Continual Relation Extraction through Prototypical Contrastive Learning](https://mlsanity.com/paper/2210.04513/) (2022)
- [Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation](https://mlsanity.com/paper/2210.04497/) (2022)
- [Lifelong Intent Detection via Multi-Strategy Rebalancing](https://mlsanity.com/paper/2108.04445/) (2021)
- [Improving Continual Relation Extraction by Distinguishing Analogous Semantics](https://mlsanity.com/paper/2305.06620/) (2023)

## Built on by (most cited first, 7 of 7)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning](https://mlsanity.com/paper/2510.10071/) (2025)
- [Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors](https://mlsanity.com/paper/2402.15713/) (2024)
- [Constructing Enhanced Mutual Information for Online Class-Incremental Learning](https://mlsanity.com/paper/2407.18526/) (2024)
- [Information-Theoretic Dual Memory System for Continual Learning](https://mlsanity.com/paper/2501.07382/) (2025)
- [Information-Theoretic Complementary Prompts for Improved Continual Text Classification](https://mlsanity.com/paper/2505.20933/) (2025)
- [Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction](https://mlsanity.com/paper/2510.15422/) (2025)

## Abstract (arXiv)

Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks. We focus on continual text classification under the class-incremental setting. Recent CL studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. In this paper, through an in-depth exploration of the representation learning process in CL, we discover that the compression effect of the information bottleneck leads to confusion on analogous classes. To enable the model learn more sufficient representations, we propose a novel replay-based continual text classification method, InfoCL. Our approach utilizes fast-slow and current-past contrastive learning to perform mutual information maximization and better recover the previously learned representations. In addition, InfoCL incorporates an adversarial memory augmentation strategy to alleviate the overfitting problem of replay. Experimental results demonstrate that InfoCL effectively mitigates forgetting and achieves state-of-the-art performance on three text classification tasks. The code is publicly available at https://github.com/Yifan-Song793/InfoCL.