# Continual Learning of Long Topic Sequences in Neural Information Retrieval

arXiv: https://arxiv.org/abs/2201.03356 · HTML: https://mlsanity.com/paper/2201.03356/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 10 Jan 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [Efficient Continual Learning with Modular Networks and Task-Driven Priors](https://mlsanity.com/paper/2012.12631/) (2020)
- [End-to-End Neural Ad-hoc Ranking with Kernel Pooling](https://mlsanity.com/paper/1706.06613/) (2017)
- [CEDR: Contextualized Embeddings for Document Ranking](https://mlsanity.com/paper/1904.07094/) (2019)
- [SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval](https://mlsanity.com/paper/2009.13013/) (2020)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017)
- [PACRR: A Position-Aware Neural IR Model for Relevance Matching](https://mlsanity.com/paper/1704.03940/) (2017)
- [Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering](https://mlsanity.com/paper/1904.06652/) (2019)
- [Toward Continual Learning for Conversational Agents](https://mlsanity.com/paper/1712.09943/) (2017)
- [Deep Relevance Ranking Using Enhanced Document-Query Interactions](https://mlsanity.com/paper/1809.01682/) (2018)
- [Progressive Memory Banks for Incremental Domain Adaptation](https://mlsanity.com/paper/1811.00239/) (2018)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)

## Built on by (most cited first, 4 of 4)

- [Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluation](https://mlsanity.com/paper/2308.08378/) (2023)
- [Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models](https://mlsanity.com/paper/2506.00037/) (2025)
- [PromptDSI: Prompt-based Rehearsal-free Continual Learning for Document Retrieval](https://mlsanity.com/paper/2406.12593/) (2024)
- [CREAM: Continual Retrieval on Dynamic Streaming Corpora with Adaptive Soft Memory](https://mlsanity.com/paper/2601.02708/) (2026)

## Abstract (arXiv)

In information retrieval (IR) systems, trends and users' interests may change over time, altering either the distribution of requests or contents to be recommended. Since neural ranking approaches heavily depend on the training data, it is crucial to understand the transfer capacity of recent IR approaches to address new domains in the long term. In this paper, we first propose a dataset based upon the MSMarco corpus aiming at modeling a long stream of topics as well as IR property-driven controlled settings. We then in-depth analyze the ability of recent neural IR models while continually learning those streams. Our empirical study highlights in which particular cases catastrophic forgetting occurs (e.g., level of similarity between tasks, peculiarities on text length, and ways of learning models) to provide future directions in terms of model design.