mlsanityData to 7 Oct 2026

Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation

arXiv 2104.05965 ↗First posted 13 Apr 2021cs.CV cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

In this work, we address the issues of missing modalities that have arisen from the Visual Question Answer-Difference prediction task and find a novel method to solve the task at hand. We address the missing modality-the ground truth answers-that are not present at test time and use a privileged knowledge distillation scheme to deal with the issue of the missing modality. In order to efficiently do so, we first introduce a model, the "Big" Teacher, that takes the image/question/answer triplet as its input and outperforms the baseline, then use a combination of models to distill knowledge to a target network (student) that only takes the image/question pair as its inputs. We experiment our models on the VizWiz and VQA-V2 Answer Difference datasets and show through extensive experimentation and ablation the performances of our method and a diverse possibility for future research.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

29 of 38 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  10. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  11. 2018Bilinear Attention Networkscited by 277
  12. 2015Stacked Attention Networks for Image Question Answeringcited by 273
17 more
  1. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  2. 2015Unifying distillation and privileged informationcited by 218
  3. 2015Cross Modal Distillation for Supervision Transfercited by 173
  4. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  5. 2015Compact Bilinear Poolingcited by 97
  6. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  7. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  8. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  9. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  10. 2015Ask Your Neurons: A Neural-based Approach to Answering Questions about Imagescited by 51
  11. 2019Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioningcited by 35
  12. 2017Graph Distillation for Action Detection with Privileged Modalitiescited by 34
  13. 2019Why Does a Visual Question Have Different Answers?cited by 34
  14. 2019Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approachcited by 30
  15. 2019DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognitioncited by 30
  16. 2017High-Order Attention Models for Visual Question Answeringcited by 22
  17. 2018Disjoint Multi-task Learning between Heterogeneous Human-centric Taskscited by 16

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 5; 2023: 1; 2024: 1; 2025: 2.

14 papers.

Sort:

  1. 2021DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learningcited by 51
  2. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  3. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  4. 2021MCDAL: Maximum Classifier Discrepancy for Active Learningcited by 18
  5. 2024Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learningcited by 16
  6. 2023Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentationcited by 14
  7. 2021Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillationcited by 13
  8. 2022Generative Bias for Robust Visual Question Answeringcited by 12
  9. 2022Visible-Infrared Person Re-Identification Using Privileged Intermediate Informationcited by 12
  10. 2021ACP++: Action Co-occurrence Priors for Human-Object Interaction Detectioncited by 11
  11. 2022Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognitioncited by 8
  12. 2021Single-Modal Entropy based Active Learning for Visual Question Answeringcited by 6
2 more
  1. 2025Disentangling and Generating Modalities for Recommendation in Missing Modality Scenarioscited by 4
  2. 2025FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysiscited by 0