# Consistent Representation Learning for Continual Relation Extraction

arXiv: https://arxiv.org/abs/2203.02721 · HTML: https://mlsanity.com/paper/2203.02721/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 5 Mar 2022.

## Placed in

- Knowledge distillation / [Distillation for continual learning](https://mlsanity.com/distill/distillation-for-continual-learning/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Curriculum-Meta Learning for Order-Robust Continual Relation Extraction](https://mlsanity.com/paper/2101.01926/) (2021)
- [Hybrid Discriminative-Generative Training via Contrastive Learning](https://mlsanity.com/paper/2007.09070/) (2020)

## Built on by (most cited first, 28 of 28)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection](https://mlsanity.com/paper/2204.07275/) (2022)
- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [ConPET: Continual Parameter-Efficient Tuning for Large Language Models](https://mlsanity.com/paper/2309.14763/) (2023)
- [Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation](https://mlsanity.com/paper/2210.04497/) (2022)
- [Rationale-Enhanced Language Models are Better Continual Relation Learners](https://mlsanity.com/paper/2310.06547/) (2023)
- [Construction of Knowledge Graphs: State and Challenges](https://mlsanity.com/paper/2302.11509/) (2023)
- [Class-Incremental Learning based on Label Generation](https://mlsanity.com/paper/2306.12619/) (2023)
- [Improving Continual Relation Extraction by Distinguishing Analogous Semantics](https://mlsanity.com/paper/2305.06620/) (2023)
- [Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction](https://mlsanity.com/paper/2305.16559/) (2023)
- [Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach](https://mlsanity.com/paper/2209.00243/) (2022)
- [Enhancing Continual Relation Extraction via Classifier Decomposition](https://mlsanity.com/paper/2305.04636/) (2023)
- [InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective](https://mlsanity.com/paper/2310.06362/) (2023)
- [ELICIT: LLM Augmentation via External In-Context Capability](https://mlsanity.com/paper/2410.09343/) (2024)
- [Serial Contrastive Knowledge Distillation for Continual Few-shot Relation Extraction](https://mlsanity.com/paper/2305.06616/) (2023)
- [Towards Incremental Learning in Large Language Models: A Critical Review](https://mlsanity.com/paper/2404.18311/) (2024)
- [Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective](https://mlsanity.com/paper/2412.08285/) (2024)
- [Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion](https://mlsanity.com/paper/2305.12270/) (2023)
- [Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors](https://mlsanity.com/paper/2402.15713/) (2024)
- [Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild](https://mlsanity.com/paper/2305.07085/) (2023)
- [Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Cases](https://mlsanity.com/paper/2508.12031/) (2025)
- [DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation](https://mlsanity.com/paper/2403.02718/) (2024)
- [Lifelong Event Detection via Optimal Transport](https://mlsanity.com/paper/2410.08905/) (2024)
- [RepCL: Exploring Effective Representation for Continual Text Classification](https://mlsanity.com/paper/2305.07289/) (2023)
- [Open-world Semi-supervised Generalized Relation Discovery Aligned in a Real-world Setting](https://mlsanity.com/paper/2305.13533/) (2023)
- [WAVE++: Capturing Within-Task Variance for Continual Relation Extraction with Adaptive Prompting](https://mlsanity.com/paper/2505.13944/) (2025)
- [Post-Training Language Models for Continual Relation Extraction](https://mlsanity.com/paper/2504.05214/) (2025)
- [Repurposing Adversarial Perturbations for Continual Learning: From Defense to Active Alignment](https://mlsanity.com/paper/2606.02322/) (2026)

## Abstract (arXiv)

Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and replaying them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on imbalanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when replaying memory. Specifically, supervised contrastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation representation. Then, contrastive replay is conducted of the samples in memory and makes the model retain the knowledge of historical relations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn consistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset.