# VCD: Visual Causality Discovery for Cross-Modal Question Reasoning

arXiv: https://arxiv.org/abs/2304.08083 · HTML: https://mlsanity.com/paper/2304.08083/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 83 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation](https://mlsanity.com/paper/2007.11978/) (2020)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)
- [Grounding Visual Explanations](https://mlsanity.com/paper/1807.09685/) (2018)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [Causal Attention for Unbiased Visual Recognition](https://mlsanity.com/paper/2108.08782/) (2021)

## Built on by (most cited first, 2 of 2)

- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)

## Abstract (arXiv)

Existing visual question reasoning methods usually fail to explicitly discover the inherent causal mechanism and ignore jointly modeling cross-modal event temporality and causality. In this paper, we propose a visual question reasoning framework named Cross-Modal Question Reasoning (CMQR), to discover temporal causal structure and mitigate visual spurious correlation by causal intervention. To explicitly discover visual causal structure, the Visual Causality Discovery (VCD) architecture is proposed to find question-critical scene temporally and disentangle the visual spurious correlations by attention-based front-door causal intervention module named Local-Global Causal Attention Module (LGCAM). To align the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build an Interactive Visual-Linguistic Transformer (IVLT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. Extensive experiments on four datasets demonstrate the superiority of CMQR for discovering visual causal structures and achieving robust question reasoning.