# Visual Commonsense based Heterogeneous Graph Contrastive Learning

arXiv: https://arxiv.org/abs/2311.06553 · HTML: https://mlsanity.com/paper/2311.06553/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Builds on (most cited first, 30 of 30)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval](https://mlsanity.com/paper/2104.10054/) (2021)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [DeepStory: Video Story QA by Deep Embedded Memory Networks](https://mlsanity.com/paper/1707.00836/) (2017)
- [Consensus-Aware Visual-Semantic Embedding for Image-Text Matching](https://mlsanity.com/paper/2007.08883/) (2020)
- [Learning Visual Commonsense for Robust Scene Graph Generation](https://mlsanity.com/paper/2006.09623/) (2020)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering](https://mlsanity.com/paper/2106.13432/) (2021)
- [Self-supervised pre-training and contrastive representation learning for multiple-choice video QA](https://mlsanity.com/paper/2009.08043/) (2020)
- [A Read-Write Memory Network for Movie Story Understanding](https://mlsanity.com/paper/1709.09345/) (2017)
- [Context-Aware Group Captioning via Self-Attention and Contrastive Features](https://mlsanity.com/paper/2004.03708/) (2020)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)

## Built on by (most cited first, 1 of 1)

- [VIKSER: Visual Knowledge-Driven Self-Reinforcing Reasoning Framework](https://mlsanity.com/paper/2502.00711/) (2025)

## Abstract (arXiv)

How to select relevant key objects and reason about the complex relationships cross vision and linguistic domain are two key issues in many multi-modality applications such as visual question answering (VQA). In this work, we incorporate the visual commonsense information and propose a heterogeneous graph contrastive learning method to better finish the visual reasoning task. Our method is designed as a plug-and-play way, so that it can be quickly and easily combined with a wide range of representative methods. Specifically, our model contains two key components: the Commonsense-based Contrastive Learning and the Graph Relation Network. Using contrastive learning, we guide the model concentrate more on discriminative objects and relevant visual commonsense attributes. Besides, thanks to the introduction of the Graph Relation Network, the model reasons about the correlations between homogeneous edges and the similarities between heterogeneous edges, which makes information transmission more effective. Extensive experiments on four benchmarks show that our method greatly improves seven representative VQA models, demonstrating its effectiveness and generalizability.