mlsanityData to 7 Oct 2026

Studying Catastrophic Forgetting in Neural Ranking Models

arXiv 2101.06984 ↗First posted 18 Jan 2021cs.IR cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Several deep neural ranking models have been proposed in the recent IR literature. While their transferability to one target domain held by a dataset has been widely addressed using traditional domain adaptation strategies, the question of their cross-domain transferability is still under-studied. We study here in what extent neural ranking models catastrophically forget old knowledge acquired from previously observed domains after acquiring new knowledge, leading to performance decrease on those domains. Our experiments show that the effectiveness of neuralIR ranking models is achieved at the cost of catastrophic forgetting and that a lifelong learning strategy using a cross-domain regularizer success-fully mitigates the problem. Using an explanatory approach built on a regression model, we also show the effect of domain characteristics on the rise of catastrophic forgetting. We believe that the obtained results can be useful for both theoretical and practical future work in neural IR.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

38 of 54 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2014How transferable are features in deep neural networks?cited by 1,216
  9. 2020In Search of Lost Domain Generalizationcited by 1,042
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  11. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  12. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
26 more
  1. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  2. 2019Episodic Memory in Lifelong Language Learningcited by 188
  3. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  4. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  5. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  6. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 113
  7. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  8. 2017End-to-End Neural Ad-hoc Ranking with Kernel Poolingcited by 85
  9. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  10. 2017Neural Ranking Models with Weak Supervisioncited by 61
  11. 2020Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directionscited by 34
  12. 2017Neural Domain Adaptation for Biomedical Question Answeringcited by 30
  13. 2019Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Modelscited by 24
  14. 2016How Transferable are Neural Networks in NLP Applications?cited by 23
  15. 2017PACRR: A Position-Aware Neural IR Model for Relevance Matchingcited by 22
  16. 2019Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answeringcited by 21
  17. 2018Progressive Memory Banks for Incremental Domain Adaptationcited by 12
  18. 2019An Updated Duet Model for Passage Re-rankingcited by 12
  19. 2018Cross Domain Regularization for Neural Ranking Models Using Adversarial Learningcited by 10
  20. 2018Lifelong Domain Word Embedding via Meta-Learningcited by 8
  21. 2018Overcoming catastrophic forgetting problem by weight consolidation and long-term memorycited by 6
  22. 2019On the Effect of Low-Frequency Terms on Neural-IR Modelscited by 6
  23. 2016Learning through Dialogue Interactions by Asking Questionscited by 5
  24. 2018Towards a Continuous Knowledge Learning Engine for Chatbotscited by 4
  25. 2020Adaptive Parameterization for Neural Dialogue Generationcited by 3
  26. 2020Does Data Augmentation Improve Generalization in NLP?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3; 2026: 2.

9 papers.

  1. 2024Scaling Sequential Recommendation Models with Transformerscited by 7
  2. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  3. 2025Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Modelscited by 5
  4. 2022Continual Learning of Long Topic Sequences in Neural Information Retrievalcited by 4
  5. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
  6. 2026When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agentscited by 2
  7. 2025MURR: Model Updating with Regularized Replay for Searching a Document Streamcited by 0
  8. 2025Optimisation Is Not What You Needcited by 0
  9. 2026Embedding Surgery: Localized Updates for Adaptive Ranking Correction in Dense Retrievalcited by 0