# Debiasing Multimodal Sarcasm Detection with Contrastive Learning

arXiv: https://arxiv.org/abs/2312.10493 · HTML: https://mlsanity.com/paper/2312.10493/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [OpenICL: An Open-Source Framework for In-context Learning](https://mlsanity.com/paper/2303.02913/) (2023)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1610.08815/) (2016)
- [Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement](https://mlsanity.com/paper/2210.03501/) (2022)
- [FiLMing Multimodal Sarcasm Detection with Attention](https://mlsanity.com/paper/2110.00416/) (2021)

## Built on by (most cited first, 13 of 13)

- [D-HUMOR: Dark Humor Understanding via Multimodal Open-ended Reasoning -- A Benchmark Dataset and Method](https://mlsanity.com/paper/2509.06771/) (2025)
- [Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue](https://mlsanity.com/paper/2402.03658/) (2024)
- [RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.13008/) (2024)
- [Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models](https://mlsanity.com/paper/2503.12149/) (2025)
- [IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection](https://mlsanity.com/paper/2505.16258/) (2025)
- [Optimism, Expectation, or Sarcasm? Multi-Class Hope Speech Detection in Spanish and English](https://mlsanity.com/paper/2504.17974/) (2025)
- [Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2506.19420/) (2025)
- [ViSP: A PPO-Driven Framework for Sarcasm Generation with Contrastive Learning](https://mlsanity.com/paper/2507.09482/) (2025)
- [Target-oriented Multimodal Sentiment Classification with Counterfactual-enhanced Debiasing](https://mlsanity.com/paper/2509.09160/) (2025)
- [Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text](https://mlsanity.com/paper/2602.21933/) (2026)
- [PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention](https://mlsanity.com/paper/2605.02447/) (2026)
- [Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2608.19942/) (2026)
- [Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2610.17707/) (2026)

## Abstract (arXiv)

Despite commendable achievements made by existing work, prevailing multimodal sarcasm detection studies rely more on textual content over visual information. It unavoidably induces spurious correlations between textual words and labels, thereby significantly hindering the models' generalization capability. To address this problem, we define the task of out-of-distribution (OOD) multimodal sarcasm detection, which aims to evaluate models' generalizability when the word distribution is different in training and testing settings. Moreover, we propose a novel debiasing multimodal sarcasm detection framework with contrastive learning, which aims to mitigate the harmful effect of biased textual factors for robust OOD generalization. In particular, we first design counterfactual data augmentation to construct the positive samples with dissimilar word biases and negative samples with similar word biases. Subsequently, we devise an adapted debiasing contrastive learning mechanism to empower the model to learn robust task-relevant features and alleviate the adverse effect of biased words. Extensive experiments show the superiority of the proposed framework.