mlsanityData to 7 Oct 2026

Multimodal Commonsense Knowledge Distillation for Visual Question Answering

arXiv 2411.02722 ↗First posted 5 Nov 2024cs.CL cs.AIcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 13 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

9 of 13 references.

References · 9

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  5. 2018Graph Convolutional Networks for Text Classificationcited by 345
  6. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  7. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  8. 2020VICTR: Visual Information Captured Text Representation for Text-to-Image Multimodal Taskscited by 12
  9. 2022Understanding Graph Convolutional Networks for Text Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 3.

11 papers.

  1. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  2. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  3. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 9
  4. 2025LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loopscited by 9
  5. 2025A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planningcited by 4
  6. 2025Cognitive-Inspired Hierarchical Attention Fusion With Visual and Textual for Cross-Domain Sequential Recommendationcited by 2
  7. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 1
  8. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  9. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1
  10. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMscited by 1
  11. 2026Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasonerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Visual Commonsense based Heterogeneous Graph Contrastive Learning
  2. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering
  3. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
  4. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge
  5. 2026KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
  6. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  7. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  8. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison