# Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

arXiv: https://arxiv.org/abs/2207.11652 · HTML: https://mlsanity.com/paper/2207.11652/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 24 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Multi-attention Recurrent Network for Human Communication Comprehension](https://mlsanity.com/paper/1802.00923/) (2018)
- [Contrastive Learning for Cold-Start Recommendation](https://mlsanity.com/paper/2107.05315/) (2021)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue](https://mlsanity.com/paper/2009.09945/) (2020)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation](https://mlsanity.com/paper/2108.01682/) (2021)
- [TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2009.02902/) (2020)
- [Interest-aware Message-Passing GCN for Recommendation](https://mlsanity.com/paper/2102.10044/) (2021)
- [Efficient Graph Deep Learning in TensorFlow with tf_geometric](https://mlsanity.com/paper/2101.11552/) (2021)
- [Empowering Language Understanding with Counterfactual Reasoning](https://mlsanity.com/paper/2106.03046/) (2021)
- [A Neural, Interactive-predictive System for Multimodal Sequence to Sequence Tasks](https://mlsanity.com/paper/1905.08181/) (2019)

## Built on by (most cited first, 18 of 18)

- [Towards Multimodal Sentiment Analysis Debiasing via Bias Purification](https://mlsanity.com/paper/2403.05023/) (2024)
- [Discriminative Probing and Tuning for Text-to-Image Generation](https://mlsanity.com/paper/2403.04321/) (2024)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [General Debiasing for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2307.10511/) (2023)
- [Multi-queue Momentum Contrast for Microvideo-Product Retrieval](https://mlsanity.com/paper/2212.11471/) (2022)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts](https://mlsanity.com/paper/2506.10452/) (2025)
- [Boosting Temporal Sentence Grounding via Causal Inference](https://mlsanity.com/paper/2507.04958/) (2025)
- [Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2210.15824/) (2022)
- [Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification](https://mlsanity.com/paper/2408.11237/) (2024)
- [Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.04999/) (2025)
- [Towards Minimal Causal Representations for Human Multimodal Language Understanding](https://mlsanity.com/paper/2509.21805/) (2025)
- [Calibrating Multimodal Consensus for Emotion Recognition](https://mlsanity.com/paper/2510.20256/) (2025)
- [Multi-Modal Image Fusion via Intervention-Stable Feature Learning](https://mlsanity.com/paper/2603.23272/) (2026)
- [Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective](https://mlsanity.com/paper/2604.18460/) (2026)
- [Subgraph Generation for Generalizing on Out-of-Distribution Links](https://mlsanity.com/paper/2507.11710/) (2025)
- [MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis](https://mlsanity.com/paper/2512.07430/) (2025)
- [MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues](https://mlsanity.com/paper/2601.20451/) (2026)

## Abstract (arXiv)

Existing studies on multimodal sentiment analysis heavily rely on textual modality and unavoidably induce the spurious correlations between textual words and sentiment labels. This greatly hinders the model generalization ability. To address this problem, we define the task of out-of-distribution (OOD) multimodal sentiment analysis. This task aims to estimate and mitigate the bad effect of textual modality for strong OOD generalization. To this end, we embrace causal inference, which inspects the causal relationships via a causal graph. From the graph, we find that the spurious correlations are attributed to the direct effect of textual modality on the model prediction while the indirect one is more reliable by considering multimodal semantics. Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis, which captures the direct effect of textual modality via an extra text model and estimates the indirect one by a multimodal model. During the inference, we first estimate the direct effect by the counterfactual inference, and then subtract it from the total effect of all modalities to obtain the indirect effect for reliable prediction. Extensive experiments show the superior effectiveness and generalization ability of our proposed framework.