# Towards Deconfounded Image-Text Matching with Causal Inference

arXiv: https://arxiv.org/abs/2408.12292 · HTML: https://mlsanity.com/paper/2408.12292/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2003.03772/) (2020)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://mlsanity.com/paper/1909.05506/) (2019)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014)
- [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Consensus-Aware Visual-Semantic Embedding for Image-Text Matching](https://mlsanity.com/paper/2007.08883/) (2020)
- [Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching](https://mlsanity.com/paper/1909.11416/) (2019)
- [DeVLBert: Learning Deconfounded Visio-Linguistic Representations](https://mlsanity.com/paper/2008.06884/) (2020)
- [Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval](https://mlsanity.com/paper/2210.08908/) (2022)
- [Towards Unbiased Visual Emotion Recognition via Causal Intervention](https://mlsanity.com/paper/2107.12096/) (2021)
- [Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching](https://mlsanity.com/paper/2104.10406/) (2021)

## Built on by (most cited first, 6 of 6)

- [INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval](https://mlsanity.com/paper/2604.18051/) (2026)
- [CWNet: Causal Wavelet Network for Low-Light Image Enhancement](https://mlsanity.com/paper/2507.10689/) (2025)
- [Fair Deepfake Detectors Can Generalize](https://mlsanity.com/paper/2507.02645/) (2025)
- [Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models](https://mlsanity.com/paper/2505.19474/) (2025)
- [Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching](https://mlsanity.com/paper/2507.09256/) (2025)
- [CIVQLLIE: Causal Intervention with Vector Quantization for Low-Light Image Enhancement](https://mlsanity.com/paper/2508.03338/) (2025)

## Abstract (arXiv)

Prior image-text matching methods have shown remarkable performance on many benchmark datasets, but most of them overlook the bias in the dataset, which exists in intra-modal and inter-modal, and tend to learn the spurious correlations that extremely degrade the generalization ability of the model. Furthermore, these methods often incorporate biased external knowledge from large-scale datasets as prior knowledge into image-text matching model, which is inevitable to force model further learn biased associations. To address above limitations, this paper firstly utilizes Structural Causal Models (SCMs) to illustrate how intra- and inter-modal confounders damage the image-text matching. Then, we employ backdoor adjustment to propose an innovative Deconfounded Causal Inference Network (DCIN) for image-text matching task. DCIN (1) decomposes the intra- and inter-modal confounders and incorporates them into the encoding stage of visual and textual features, effectively eliminating the spurious correlations during image-text matching, and (2) uses causal inference to mitigate biases of external knowledge. Consequently, the model can learn causality instead of spurious correlations caused by dataset bias. Extensive experiments on two well-known benchmark datasets, i.e., Flickr30K and MSCOCO, demonstrate the superiority of our proposed method.