# Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering

arXiv: https://arxiv.org/abs/1906.04229 · HTML: https://mlsanity.com/paper/1906.04229/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 33)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks](https://mlsanity.com/paper/2206.09059/) (2022)
- [Lifelong Language Pretraining with Distribution-Specialized Experts](https://mlsanity.com/paper/2305.12281/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Recent Advances of Multimodal Continual Learning: A Comprehensive Survey](https://mlsanity.com/paper/2410.05352/) (2024)
- [Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task](https://mlsanity.com/paper/2208.12037/) (2022)
- [Continual Training of Language Models for Few-Shot Learning](https://mlsanity.com/paper/2210.05549/) (2022)
- [AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models](https://mlsanity.com/paper/2102.05126/) (2021)
- [Machine Psychology](https://mlsanity.com/paper/2303.13988/) (2023)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [Selective Replay Enhances Learning in Online Continual Analogical Reasoning](https://mlsanity.com/paper/2103.03987/) (2021)
- [Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering](https://mlsanity.com/paper/2502.04469/) (2025)
- [Efficient Task-Oriented Dialogue Systems with Response Selection as an Auxiliary Task](https://mlsanity.com/paper/2208.07097/) (2022)
- [I2I: Initializing Adapters with Improvised Knowledge](https://mlsanity.com/paper/2304.02168/) (2023)
- [Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation](https://mlsanity.com/paper/2406.19297/) (2024)
- [CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering](https://mlsanity.com/paper/2211.10567/) (2022)
- [One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering](https://mlsanity.com/paper/2411.02210/) (2024)
- [Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering](https://mlsanity.com/paper/2210.00044/) (2022)
- [Visually Grounded Continual Language Learning with Selective Specialization](https://mlsanity.com/paper/2310.15571/) (2023)
- [Affordance-First Decomposition for Continual Learning in Video-Language Understanding](https://mlsanity.com/paper/2512.00694/) (2025)
- [HOP to the Next Tasks and Domains for Continual Learning in NLP](https://mlsanity.com/paper/2402.18449/) (2024)
- [Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation](https://mlsanity.com/paper/2109.06308/) (2021)
- [Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model](https://mlsanity.com/paper/2207.06839/) (2022)
- [A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)](https://mlsanity.com/paper/2406.19054/) (2024)
- [CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models](https://mlsanity.com/paper/2510.13008/) (2025)
- [Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs](https://mlsanity.com/paper/2606.23063/) (2026)

## Abstract (arXiv)

We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA). Motivated by evidence from psycholinguistics, we devise a set of linguistically-informed VQA tasks, which differ by the types of questions involved (Wh-questions and polar questions). We test what impact task difficulty has on continual learning, and whether the order in which a child acquires question types facilitates computational models. Our results show that dramatic forgetting is at play and that task difficulty and order matter. Two well-known current continual learning methods mitigate the problem only to a limiting degree.