# Heterogeneous Graph Learning for Visual Commonsense Reasoning

arXiv: https://arxiv.org/abs/1910.11475 · HTML: https://mlsanity.com/paper/1910.11475/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 13 of 13)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [Joint Answering and Explanation for Visual Commonsense Reasoning](https://mlsanity.com/paper/2202.12626/) (2022)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)
- [Multi-modal Large Language Model Enhanced Pseudo 3D Perception Framework for Visual Commonsense Reasoning](https://mlsanity.com/paper/2301.13335/) (2023)
- [Learning to Agree on Vision Attention for Visual Commonsense Reasoning](https://mlsanity.com/paper/2302.02117/) (2023)
- [EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning](https://mlsanity.com/paper/2404.13847/) (2024)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [Causal Debiasing for Visual Commonsense Reasoning](https://mlsanity.com/paper/2510.20281/) (2025)

## Abstract (arXiv)

Visual commonsense reasoning task aims at leading the research field into solving cognition-level reasoning with the ability of predicting correct answers and meanwhile providing convincing reasoning paths, resulting in three sub-tasks i.e., Q->A, QA->R and Q->AR. It poses great challenges over the proper semantic alignment between vision and linguistic domains and knowledge reasoning to generate persuasive reasoning paths. Existing works either resort to a powerful end-to-end network that cannot produce interpretable reasoning paths or solely explore intra-relationship of visual objects (homogeneous graph) while ignoring the cross-domain semantic alignment among visual concepts and linguistic words. In this paper, we propose a new Heterogeneous Graph Learning (HGL) framework for seamlessly integrating the intra-graph and inter-graph reasoning in order to bridge vision and language domain. Our HGL consists of a primal vision-to-answer heterogeneous graph (VAHG) module and a dual question-to-answer heterogeneous graph (QAHG) module to interactively refine reasoning paths for semantic agreement. Moreover, our HGL integrates a contextual voting module to exploit a long-range visual context for better global reasoning. Experiments on the large-scale Visual Commonsense Reasoning benchmark demonstrate the superior performance of our proposed modules on three tasks (improving 5% accuracy on Q->A, 3.5% on QA->R, 5.8% on Q->AR)