# Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

arXiv: https://arxiv.org/abs/2207.12647 · HTML: https://mlsanity.com/paper/2207.12647/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 74 of its 121 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)

## Built on by (most cited first, 30 of 57)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method](https://mlsanity.com/paper/2412.09082/) (2024)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Abductive Ego-View Accident Video Understanding for Safe Driving Perception](https://mlsanity.com/paper/2403.00436/) (2024)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports](https://mlsanity.com/paper/2401.01505/) (2024)
- [Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answering](https://mlsanity.com/paper/2503.11117/) (2025)
- [CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process](https://mlsanity.com/paper/2401.14535/) (2024)
- [MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning](https://mlsanity.com/paper/2409.17647/) (2024)
- [Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2502.12425/) (2025)
- [AVD2: Accident Video Diffusion for Accident Video Description](https://mlsanity.com/paper/2502.14801/) (2025)
- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [Referring Atomic Video Action Recognition](https://mlsanity.com/paper/2407.01872/) (2024)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [Urban Regional Function Guided Traffic Flow Prediction](https://mlsanity.com/paper/2303.09789/) (2023)
- [Causal reasoning in typical computer vision tasks](https://mlsanity.com/paper/2307.13992/) (2023)
- [Cross-modal Causal Relation Alignment for Video Question Grounding](https://mlsanity.com/paper/2503.07635/) (2025)
- [CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillation](https://mlsanity.com/paper/2502.01312/) (2025)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [Synergistic Prompting for Robust Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2507.07802/) (2025)
- [PRVR: Partially Relevant Video Retrieval](https://mlsanity.com/paper/2208.12510/) (2022)
- [DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback](https://mlsanity.com/paper/2306.07553/) (2023)
- [Structure Causal Models and LLMs Integration in Medical Visual Question Answering](https://mlsanity.com/paper/2505.02703/) (2025)
- [CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning](https://mlsanity.com/paper/2306.17462/) (2023)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)
- [MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning](https://mlsanity.com/paper/2501.07227/) (2025)
- [VideoPure: Diffusion-based Adversarial Purification for Video Recognition](https://mlsanity.com/paper/2501.14999/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework](https://mlsanity.com/paper/2601.02783/) (2026)
- [FineCausal: A Causal-Based Framework for Interpretable Fine-Grained Action Quality Assessment](https://mlsanity.com/paper/2503.23911/) (2025)

## Abstract (arXiv)

Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named Cross-Modal Causal RelatIonal Reasoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering. The datasets, code, and models are available at https://github.com/HCPLab-SYSU/CMCIR.