# Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach

arXiv: https://arxiv.org/abs/2209.00243 · HTML: https://mlsanity.com/paper/2209.00243/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Striking the Right Balance with Uncertainty](https://mlsanity.com/paper/1901.07590/) (2019)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Adjusting Decision Boundary for Class Imbalanced Learning](https://mlsanity.com/paper/1912.01857/) (2019)
- [Continual Learning in Generative Adversarial Nets](https://mlsanity.com/paper/1705.08395/) (2017)
- [Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path](https://mlsanity.com/paper/1508.03720/) (2015)
- [Curriculum-Meta Learning for Order-Robust Continual Relation Extraction](https://mlsanity.com/paper/2101.01926/) (2021)
- [Consistent Representation Learning for Continual Relation Extraction](https://mlsanity.com/paper/2203.02721/) (2022)
- [Improving Relation Extraction by Pre-trained Language Representations](https://mlsanity.com/paper/1906.03088/) (2019)
- [Improved Relation Extraction with Feature-Rich Compositional Embedding Models](https://mlsanity.com/paper/1505.02419/) (2015)

## Built on by (most cited first, 8 of 8)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation](https://mlsanity.com/paper/2210.04497/) (2022)
- [Rationale-Enhanced Language Models are Better Continual Relation Learners](https://mlsanity.com/paper/2310.06547/) (2023)
- [Enhancing Continual Relation Extraction via Classifier Decomposition](https://mlsanity.com/paper/2305.04636/) (2023)
- [DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation](https://mlsanity.com/paper/2403.02718/) (2024)
- [RepCL: Exploring Effective Representation for Continual Text Classification](https://mlsanity.com/paper/2305.07289/) (2023)
- [DepNeCTI: Dependency-based Nested Compound Type Identification for Sanskrit](https://mlsanity.com/paper/2310.09501/) (2023)
- [Balancing the Causal Effects in Class-Incremental Learning](https://mlsanity.com/paper/2402.10063/) (2024)

## Abstract (arXiv)

Continual relation extraction (CRE) requires the model to continually learn new relations from class-incremental data streams. In this paper, we propose a Frustratingly easy but Effective Approach (FEA) method with two learning stages for CRE: 1) Fast Adaption (FA) warms up the model with only new data. 2) Balanced Tuning (BT) finetunes the model on the balanced memory data. Despite its simplicity, FEA achieves comparable (on TACRED or superior (on FewRel) performance compared with the state-of-the-art baselines. With careful examinations, we find that the data imbalance between new and old relations leads to a skewed decision boundary in the head classifiers over the pretrained encoders, thus hurting the overall performance. In FEA, the FA stage unleashes the potential of memory data for the subsequent finetuning, while the BT stage helps establish a more balanced decision boundary. With a unified view, we find that two strong CRE baselines can be subsumed into the proposed training pipeline. The success of FEA also provides actionable insights and suggestions for future model designing in CRE.