# Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors

arXiv: https://arxiv.org/abs/2402.15713 · HTML: https://mlsanity.com/paper/2402.15713/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 24 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Consistent Representation Learning for Continual Relation Extraction](https://mlsanity.com/paper/2203.02721/) (2022)
- [Improving Continual Relation Extraction through Prototypical Contrastive Learning](https://mlsanity.com/paper/2210.04513/) (2022)
- [Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation](https://mlsanity.com/paper/2210.04497/) (2022)
- [InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective](https://mlsanity.com/paper/2310.06362/) (2023)
- [Serial Contrastive Knowledge Distillation for Continual Few-shot Relation Extraction](https://mlsanity.com/paper/2305.06616/) (2023)

## Built on by (most cited first, 4 of 4)

- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [Preserving Generalization of Language models in Few-shot Continual Relation Extraction](https://mlsanity.com/paper/2410.00334/) (2024)
- [Few-Shot, No Problem: Descriptive Continual Relation Extraction](https://mlsanity.com/paper/2502.20596/) (2025)
- [Few-shot Continual Relation Extraction via Open Information Extraction](https://mlsanity.com/paper/2502.16648/) (2025)

## Abstract (arXiv)

Continual Few-shot Relation Extraction (CFRE) is a practical problem that requires the model to continuously learn novel relations while avoiding forgetting old ones with few labeled training data. The primary challenges are catastrophic forgetting and overfitting. This paper harnesses prompt learning to explore the implicit capabilities of pre-trained language models to address the above two challenges, thereby making language models better continual few-shot relation extractors. Specifically, we propose a Contrastive Prompt Learning framework, which designs prompt representation to acquire more generalized knowledge that can be easily adapted to old and new categories, and margin-based contrastive learning to focus more on hard samples, therefore alleviating catastrophic forgetting and overfitting issues. To further remedy overfitting in low-resource scenarios, we introduce an effective memory augmentation strategy that employs well-crafted prompts to guide ChatGPT in generating diverse samples. Extensive experiments demonstrate that our method outperforms state-of-the-art methods by a large margin and significantly mitigates catastrophic forgetting and overfitting in low-resource scenarios.