# MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering

arXiv: https://arxiv.org/abs/2601.01926 · HTML: https://mlsanity.com/paper/2601.01926/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Social Biases through the Text-to-Image Generation Lens](https://mlsanity.com/paper/2304.06034/) (2023)
- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Information Maximizing Visual Question Generation](https://mlsanity.com/paper/1903.11207/) (2019)
- [Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control](https://mlsanity.com/paper/1907.13322/) (2019)
- [Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task](https://mlsanity.com/paper/2208.12037/) (2022)
- [Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering](https://mlsanity.com/paper/1906.04229/) (2019)
- [Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension](https://mlsanity.com/paper/1911.00202/) (2019)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.12045/) (2023)
- [Efficient Multi-Task and Transfer Reinforcement Learning with Parameter-Compositional Framework](https://mlsanity.com/paper/2306.01839/) (2023)

## Abstract (arXiv)

Visual Question Answering (VQA) requires models to reason over multimodal information, combining visual and textual data. With the development of continual learning, significant progress has been made in retaining knowledge and adapting to new information in the VQA domain. However, current methods often struggle with balancing knowledge retention, adaptation, and robust feature representation. To address these challenges, we propose a novel framework with adaptive memory allocation and global noise filtering called MacVQA for visual question answering. MacVQA fuses visual and question information while filtering noise to ensure robust representations, and employs prototype-based memory allocation to optimize feature quality and memory usage. These designs enable MacVQA to balance knowledge acquisition, retention, and compositional generalization in continual VQA learning. Experiments on ten continual VQA tasks show that MacVQA outperforms existing baselines, achieving 43.38% average accuracy and 2.32% average forgetting on standard tasks, and 42.53% average accuracy and 3.60% average forgetting on novel composition tasks.