# Two Causal Principles for Improving Visual Dialog

arXiv: https://arxiv.org/abs/1911.10496 · HTML: https://mlsanity.com/paper/1911.10496/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2019.

## Built on by (most cited first, 30 of 75)

- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention](https://mlsanity.com/paper/2106.09233/) (2021)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Robust Emotion Recognition in Context Debiasing](https://mlsanity.com/paper/2403.05963/) (2024)
- [Learning Domain Invariant Representations for Generalizable Person Re-Identification](https://mlsanity.com/paper/2103.15890/) (2021)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [DiGeo: Discriminative Geometry-Aware Learning for Generalized Few-Shot Object Detection](https://mlsanity.com/paper/2303.09674/) (2023)
- [Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification](https://mlsanity.com/paper/2112.14380/) (2021)
- [PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts](https://mlsanity.com/paper/2305.14839/) (2023)
- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [Counterfactual Intervention Feature Transfer for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2208.00967/) (2022)
- [Selective Query-guided Debiasing for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2210.08714/) (2022)
- [The Blessings of Unlabeled Background in Untrimmed Videos](https://mlsanity.com/paper/2103.13183/) (2021)
- [Deconfounded Visual Grounding](https://mlsanity.com/paper/2112.15324/) (2021)
- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [Improving Weakly-supervised Object Localization via Causal Intervention](https://mlsanity.com/paper/2104.10351/) (2021)
- [DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue](https://mlsanity.com/paper/2101.00151/) (2021)
- [Adversarial Visual Robustness by Causal Intervention](https://mlsanity.com/paper/2106.09534/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [Interventional Multi-Instance Learning with Deconfounded Instance-Level Prediction](https://mlsanity.com/paper/2204.09204/) (2022)
- [Element Intervention for Open Relation Extraction](https://mlsanity.com/paper/2106.09558/) (2021)

## Abstract (arXiv)

This paper unravels the design tricks adopted by us, the champion team MReaL-BDAI, for Visual Dialog Challenge 2019: two causal principles for improving Visual Dialog (VisDial). By "improving", we mean that they can promote almost every existing VisDial model to the state-of-the-art performance on the leader-board. Such a major improvement is only due to our careful inspection on the causality behind the model and data, finding that the community has overlooked two causalities in VisDial. Intuitively, Principle 1 suggests: we should remove the direct input of the dialog history to the answer model, otherwise a harmful shortcut bias will be introduced; Principle 2 says: there is an unobserved confounder for history, question, and answer, leading to spurious correlations from training data. In particular, to remove the confounder suggested in Principle 2, we propose several causal intervention algorithms, which make the training fundamentally different from the traditional likelihood estimation. Note that the two principles are model-agnostic, so they are applicable in any VisDial model. The code is available at https://github.com/simpleshinobu/visdial-principles.