mlsanityData to 14 Oct 2026

Continual Learning of Long Topic Sequences in Neural Information Retrieval

arXiv 2201.03356 ↗First posted 10 Jan 2022cs.IR cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In information retrieval (IR) systems, trends and users' interests may change over time, altering either the distribution of requests or contents to be recommended. Since neural ranking approaches heavily depend on the training data, it is crucial to understand the transfer capacity of recent IR approaches to address new domains in the long term. In this paper, we first propose a dataset based upon the MSMarco corpus aiming at modeling a long stream of topics as well as IR property-driven controlled settings. We then in-depth analyze the ability of recent neural IR models while continually learning those streams. Our empirical study highlights in which particular cases catastrophic forgetting occurs (e.g., level of similarity between tasks, peculiarities on text length, and ways of learning models) to provide future directions in terms of model design.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  11. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  12. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
21 more
  1. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  2. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  3. 2019Episodic Memory in Lifelong Language Learningcited by 188
  4. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  5. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  6. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  7. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 88
  8. 2017End-to-End Neural Ad-hoc Ranking with Kernel Poolingcited by 85
  9. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  10. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  11. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  12. 2017Neural Domain Adaptation for Biomedical Question Answeringcited by 30
  13. 2017PACRR: A Position-Aware Neural IR Model for Relevance Matchingcited by 22
  14. 2019Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answeringcited by 21
  15. 2017Toward Continual Learning for Conversational Agentscited by 14
  16. 2018Deep Relevance Ranking Using Enhanced Document-Query Interactionscited by 13
  17. 2018Progressive Memory Banks for Incremental Domain Adaptationcited by 12
  18. 2021Contrastive Fine-tuning Improves Robustness for Neural Rankerscited by 12
  19. 2018Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehensioncited by 9
  20. 2021Studying Catastrophic Forgetting in Neural Ranking Modelscited by 9
  21. 2020Adaptive Parameterization for Neural Dialogue Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  2. 2025Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Modelscited by 5
  3. 2024PromptDSI: Prompt-based Rehearsal-free Continual Learning for Document Retrievalcited by 1
  4. 2026CREAM: Continual Retrieval on Dynamic Streaming Corpora with Adaptive Soft Memorycited by 1