# Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

arXiv: https://arxiv.org/abs/2210.03501 · HTML: https://mlsanity.com/paper/2210.03501/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [Reasoning with Sarcasm by Reading In-between](https://mlsanity.com/paper/1805.02856/) (2018)

## Built on by (most cited first, 27 of 27)

- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts](https://mlsanity.com/paper/2311.09580/) (2023)
- [A Survey of Multimodal Sarcasm Detection](https://mlsanity.com/paper/2410.18882/) (2024)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)
- [Interpretable Multimodal Misinformation Detection with Logic Reasoning](https://mlsanity.com/paper/2305.05964/) (2023)
- [Debiasing Multimodal Sarcasm Detection with Contrastive Learning](https://mlsanity.com/paper/2312.10493/) (2023)
- [CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models](https://mlsanity.com/paper/2405.00390/) (2024)
- [Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding](https://mlsanity.com/paper/2403.11311/) (2024)
- [Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method](https://mlsanity.com/paper/2306.16950/) (2023)
- [Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2504.15848/) (2025)
- [Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing](https://mlsanity.com/paper/2509.15361/) (2025)
- [Evaluating Picture Description Speech for Dementia Detection using Image-text Alignment](https://mlsanity.com/paper/2308.07933/) (2023)
- [InterCLIP-MEP: Interactive CLIP and Memory-Enhanced Predictor for Multi-modal Sarcasm Detection](https://mlsanity.com/paper/2406.16464/) (2024)
- [RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.13008/) (2024)
- [Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models](https://mlsanity.com/paper/2503.12149/) (2025)
- [MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery](https://mlsanity.com/paper/2402.18169/) (2024)
- [Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment](https://mlsanity.com/paper/2403.06355/) (2024)
- [MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding](https://mlsanity.com/paper/2507.04635/) (2025)
- [Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detection](https://mlsanity.com/paper/2307.00209/) (2023)
- [Dual Modality-Aware Gated Prompt Tuning for Few-Shot Multimodal Sarcasm Detection](https://mlsanity.com/paper/2507.04468/) (2025)
- [Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection](https://mlsanity.com/paper/2507.04508/) (2025)
- [GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2601.20618/) (2026)
- [An Innovative CGL-MHA Model for Sarcasm Sentiment Recognition Using the MindSpore Framework](https://mlsanity.com/paper/2411.01264/) (2024)
- [Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student](https://mlsanity.com/paper/2510.25303/) (2025)
- [Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?](https://mlsanity.com/paper/2601.06424/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)
- [Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2610.17707/) (2026)

## Abstract (arXiv)

Sarcasm is a linguistic phenomenon indicating a discrepancy between literal meanings and implied intentions. Due to its sophisticated nature, it is usually challenging to be detected from the text itself. As a result, multi-modal sarcasm detection has received more attention in both academia and industries. However, most existing techniques only modeled the atomic-level inconsistencies between the text input and its accompanying image, ignoring more complex compositions for both modalities. Moreover, they neglected the rich information contained in external knowledge, e.g., image captions. In this paper, we propose a novel hierarchical framework for sarcasm detection by exploring both the atomic-level congruity based on multi-head cross attention mechanism and the composition-level congruity based on graph neural networks, where a post with low congruity can be identified as sarcasm. In addition, we exploit the effect of various knowledge resources for sarcasm detection. Evaluation results on a public multi-modal sarcasm detection dataset based on Twitter demonstrate the superiority of our proposed model.