# Preserving Generalization of Language models in Few-shot Continual Relation Extraction

arXiv: https://arxiv.org/abs/2410.00334 · HTML: https://mlsanity.com/paper/2410.00334/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Mixture of Experts Meets Prompt-Based Continual Learning](https://mlsanity.com/paper/2405.14124/) (2024)
- [Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation](https://mlsanity.com/paper/2203.02135/) (2022)
- [Serial Contrastive Knowledge Distillation for Continual Few-shot Relation Extraction](https://mlsanity.com/paper/2305.06616/) (2023)
- [Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors](https://mlsanity.com/paper/2402.15713/) (2024)
- [Neural Text Generation: A Practical Guide](https://mlsanity.com/paper/1711.09534/) (2017)

## Built on by (most cited first, 7 of 7)

- [Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective](https://mlsanity.com/paper/2412.08285/) (2024)
- [Few-Shot, No Problem: Descriptive Continual Relation Extraction](https://mlsanity.com/paper/2502.20596/) (2025)
- [ResidualDroppath: Enhancing Feature Reuse over Residual Connections](https://mlsanity.com/paper/2411.09475/) (2024)
- [Few-shot Continual Relation Extraction via Open Information Extraction](https://mlsanity.com/paper/2502.16648/) (2025)
- [Post-Training Language Models for Continual Relation Extraction](https://mlsanity.com/paper/2504.05214/) (2025)
- [TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation](https://mlsanity.com/paper/2606.21851/) (2026)
- [Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions](https://mlsanity.com/paper/2607.22931/) (2026)

## Abstract (arXiv)

Few-shot Continual Relations Extraction (FCRE) is an emerging and dynamic area of study where models can sequentially integrate knowledge from new relations with limited labeled data while circumventing catastrophic forgetting and preserving prior knowledge from pre-trained backbones. In this work, we introduce a novel method that leverages often-discarded language model heads. By employing these components via a mutual information maximization strategy, our approach helps maintain prior knowledge from the pre-trained backbone and strategically aligns the primary classification head, thereby enhancing model performance. Furthermore, we explore the potential of Large Language Models (LLMs), renowned for their wealth of knowledge, in addressing FCRE challenges. Our comprehensive experimental results underscore the efficacy of the proposed method and offer valuable insights for future work.