# Studying Catastrophic Forgetting in Neural Ranking Models

arXiv: https://arxiv.org/abs/2101.06984 · HTML: https://mlsanity.com/paper/2101.06984/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models](https://mlsanity.com/paper/1909.11299/) (2019)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [End-to-End Neural Ad-hoc Ranking with Kernel Pooling](https://mlsanity.com/paper/1706.06613/) (2017)
- [CEDR: Contextualized Embeddings for Document Ranking](https://mlsanity.com/paper/1904.07094/) (2019)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017)
- [Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models](https://mlsanity.com/paper/1904.09171/) (2019)
- [How Transferable are Neural Networks in NLP Applications?](https://mlsanity.com/paper/1603.06111/) (2016)
- [PACRR: A Position-Aware Neural IR Model for Relevance Matching](https://mlsanity.com/paper/1704.03940/) (2017)
- [Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering](https://mlsanity.com/paper/1904.06652/) (2019)
- [Progressive Memory Banks for Incremental Domain Adaptation](https://mlsanity.com/paper/1811.00239/) (2018)
- [An Updated Duet Model for Passage Re-ranking](https://mlsanity.com/paper/1903.07666/) (2019)

## Built on by (most cited first, 9 of 9)

- [Scaling Sequential Recommendation Models with Transformers](https://mlsanity.com/paper/2412.07585/) (2024)
- [Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluation](https://mlsanity.com/paper/2308.08378/) (2023)
- [Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models](https://mlsanity.com/paper/2506.00037/) (2025)
- [Continual Learning of Long Topic Sequences in Neural Information Retrieval](https://mlsanity.com/paper/2201.03356/) (2022)
- [EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries](https://mlsanity.com/paper/2402.13372/) (2024)
- [When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents](https://mlsanity.com/paper/2609.10750/) (2026)
- [MURR: Model Updating with Regularized Replay for Searching a Document Stream](https://mlsanity.com/paper/2504.10250/) (2025)
- [Optimisation Is Not What You Need](https://mlsanity.com/paper/2507.03045/) (2025)
- [Embedding Surgery: Localized Updates for Adaptive Ranking Correction in Dense Retrieval](https://mlsanity.com/paper/2609.05110/) (2026)

## Abstract (arXiv)

Several deep neural ranking models have been proposed in the recent IR literature. While their transferability to one target domain held by a dataset has been widely addressed using traditional domain adaptation strategies, the question of their cross-domain transferability is still under-studied. We study here in what extent neural ranking models catastrophically forget old knowledge acquired from previously observed domains after acquiring new knowledge, leading to performance decrease on those domains. Our experiments show that the effectiveness of neuralIR ranking models is achieved at the cost of catastrophic forgetting and that a lifelong learning strategy using a cross-domain regularizer success-fully mitigates the problem. Using an explanatory approach built on a regression model, we also show the effect of domain characteristics on the rise of catastrophic forgetting. We believe that the obtained results can be useful for both theoretical and practical future work in neural IR.