# Multimodal Commonsense Knowledge Distillation for Visual Question Answering

arXiv: https://arxiv.org/abs/2411.02722 · HTML: https://mlsanity.com/paper/2411.02722/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 9 of 9)

- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering](https://mlsanity.com/paper/2305.03453/) (2023)
- [VICTR: Visual Information Captured Text Representation for Text-to-Image Multimodal Tasks](https://mlsanity.com/paper/2010.03182/) (2020)
- [Understanding Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2203.16060/) (2022)

## Built on by (most cited first, 11 of 11)

- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2510.15349/) (2025)
- [Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias](https://mlsanity.com/paper/2506.06280/) (2025)
- [LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loops](https://mlsanity.com/paper/2506.14493/) (2025)
- [A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planning](https://mlsanity.com/paper/2506.07236/) (2025)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2506.03197/) (2025)
- [Cognitive-Inspired Hierarchical Attention Fusion With Visual and Textual for Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2504.15085/) (2025)
- [Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache](https://mlsanity.com/paper/2511.18811/) (2025)
- [MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.15915/) (2026)
- [From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2605.02130/) (2026)
- [Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners](https://mlsanity.com/paper/2607.28336/) (2026)

## Abstract (arXiv)

Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.