# Continual Learning for Sentence Representations Using Conceptors

arXiv: https://arxiv.org/abs/1904.09187 · HTML: https://mlsanity.com/paper/1904.09187/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2019.

## Built on by (most cited first, 22 of 22)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Continual Learning of Natural Language Processing Tasks: A Survey](https://mlsanity.com/paper/2211.12701/) (2022)
- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Continual Training of Language Models for Few-Shot Learning](https://mlsanity.com/paper/2210.05549/) (2022)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection](https://mlsanity.com/paper/2308.03300/) (2023)
- [Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks](https://mlsanity.com/paper/2302.11074/) (2023)
- [AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection](https://mlsanity.com/paper/2511.13880/) (2025)
- [Conceptor-Aided Debiasing of Large Language Models](https://mlsanity.com/paper/2211.11087/) (2022)
- [Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/2212.02800/) (2022)
- [RVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learning](https://mlsanity.com/paper/2205.10857/) (2022)
- [Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics](https://mlsanity.com/paper/2210.02284/) (2022)
- [Dynamically Addressing Unseen Rumor via Continual Learning](https://mlsanity.com/paper/2104.08775/) (2021)
- [HOP to the Next Tasks and Domains for Continual Learning in NLP](https://mlsanity.com/paper/2402.18449/) (2024)
- [Conceptors for Semantic Steering](https://mlsanity.com/paper/2605.04980/) (2026)
- [Does RoBERTa Perform Better than BERT in Continual Learning: An Attention Sink Perspective](https://mlsanity.com/paper/2410.05648/) (2024)
- [Continual Learning with Global Alignment](https://mlsanity.com/paper/2205.12186/) (2022)
- [Ask Question First for Enhancing Lifelong Language Learning](https://mlsanity.com/paper/2208.08367/) (2022)
- [Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities](https://mlsanity.com/paper/2606.31058/) (2026)

## Abstract (arXiv)

Distributed representations of sentences have become ubiquitous in natural language processing tasks. In this paper, we consider a continual learning scenario for sentence representations: Given a sequence of corpora, we aim to optimize the sentence encoder with respect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence encoders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of conceptor matrices to learn corpus-dependent features. We evaluate our approach on semantic textual similarity tasks and show that our proposed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered corpora.