mlsanityData to 14 Oct 2026

Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation

arXiv 2406.19297 ↗First posted 27 Jun 2024cs.CVcited by 7

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 50 of its 75 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

50 of 75 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2016Learning without Forgettingcited by 2,496
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  12. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  13. 2019Large Scale Incremental Learningcited by 732
  14. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  15. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  16. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  17. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  18. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  19. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  20. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  21. 2018Learning without Memorizingcited by 250
  22. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  23. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  24. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  25. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  26. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 104
  27. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 93
  28. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 83
  29. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  30. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  31. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  32. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  33. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  34. 2020RATT: Recurrent Attention to Transient Tasks for Continual Image Captioningcited by 32
  35. 2021An Investigation of Replay-based Approaches for Continual Learningcited by 26
  36. 2019ContCap: A scalable framework for continual image captioningcited by 11
  37. 2022CNLL: A Semi-supervised Approach For Continual Noisy Label Learningcited by 8
  38. 2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answeringcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  2. 2025How to Teach Large Multimodal Models New Skillscited by 11
  3. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  4. 2026Same Answer, Different Representations: Hidden instability in VLMscited by 4
  5. 2025MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answeringcited by 1
  6. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  7. 2026Regularizing modality contribution drift in multimodal continual learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  2. 2024CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering
  3. 2026An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering
  4. 2026AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning
  5. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
  6. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  7. 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
  8. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation