# Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

arXiv: https://arxiv.org/abs/1911.00202 · HTML: https://mlsanity.com/paper/1911.00202/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 27 of 27)

- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation](https://mlsanity.com/paper/2107.02408/) (2021)
- [Watermarking Graph Neural Networks based on Backdoor Attacks](https://mlsanity.com/paper/2110.11024/) (2021)
- [Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling](https://mlsanity.com/paper/2407.02446/) (2024)
- [Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks](https://mlsanity.com/paper/2402.13482/) (2024)
- [MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters](https://mlsanity.com/paper/2303.07354/) (2023)
- [Leveraging Graph Structures to Detect Hallucinations in Large Language Models](https://mlsanity.com/paper/2407.04485/) (2024)
- [Rumour Detection via Zero-shot Cross-lingual Transfer Learning](https://mlsanity.com/paper/2109.12773/) (2021)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [Domain Adaptation for Time series Transformers using One-step fine-tuning](https://mlsanity.com/paper/2401.06524/) (2024)
- [AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks](https://mlsanity.com/paper/2405.13580/) (2024)
- [LLMLagBench: Identifying Temporal Training Boundaries in Large Language Models](https://mlsanity.com/paper/2511.12116/) (2025)
- [Integrating question answering and text-to-SQL in Portuguese](https://mlsanity.com/paper/2202.04048/) (2022)
- [RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use](https://mlsanity.com/paper/2211.08192/) (2022)
- [PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models](https://mlsanity.com/paper/2305.04673/) (2023)
- [Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization](https://mlsanity.com/paper/2502.12672/) (2025)
- [FIESTA: Fisher Information-based Efficient Selective Test-time Adaptation](https://mlsanity.com/paper/2503.23257/) (2025)
- [Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation](https://mlsanity.com/paper/2109.06308/) (2021)
- [Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking](https://mlsanity.com/paper/2111.06012/) (2021)
- [Task Transfer and Domain Adaptation for Zero-Shot Question Answering](https://mlsanity.com/paper/2206.06705/) (2022)
- [Sit Back and Relax: Learning to Drive Incrementally in All Weather Conditions](https://mlsanity.com/paper/2305.18953/) (2023)
- [Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies](https://mlsanity.com/paper/2309.06089/) (2023)
- [Harnessing the Power of Multi-Lingual Datasets for Pre-training: Towards Enhancing Text Spotting Performance](https://mlsanity.com/paper/2310.00917/) (2023)
- [Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments](https://mlsanity.com/paper/2506.21129/) (2025)
- [Catastrophic Forgetting in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2511.12828/) (2025)
- [MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering](https://mlsanity.com/paper/2601.01926/) (2026)

## Abstract (arXiv)

The creation of large-scale open domain reading comprehension data sets in recent years has enabled the development of end-to-end neural comprehension models with promising results. To use these models for domains with limited training data, one of the most effective approach is to first pretrain them on large out-of-domain source data and then fine-tune them with the limited target data. The caveat of this is that after fine-tuning the comprehension models tend to perform poorly in the source domain, a phenomenon known as catastrophic forgetting. In this paper, we explore methods that overcome catastrophic forgetting during fine-tuning without assuming access to data from the source domain. We introduce new auxiliary penalty terms and observe the best performance when a combination of auxiliary penalty terms is used to regularise the fine-tuning process for adapting comprehension models. To test our methods, we develop and release 6 narrow domain data sets that could potentially be used as reading comprehension benchmarks.