# Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation

arXiv: https://arxiv.org/abs/2112.04873 · HTML: https://mlsanity.com/paper/2112.04873/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [VinVL: Revisiting Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2101.00529/) (2021)
- [e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks](https://mlsanity.com/paper/2105.03761/) (2021)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [iSarcasm: A Dataset of Intended Sarcasm](https://mlsanity.com/paper/1911.03123/) (2019)
- [Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations](https://mlsanity.com/paper/2105.09984/) (2021)
- [CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://mlsanity.com/paper/1805.06413/) (2018)
- [Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation](https://mlsanity.com/paper/1704.06836/) (2017)
- [Are Safer Looking Neighborhoods More Lively? A Multimodal Investigation into Urban Life](https://mlsanity.com/paper/1608.00462/) (2016)

## Built on by (most cited first, 25 of 25)

- [We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?](https://mlsanity.com/paper/2407.01284/) (2024)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2306.16650/) (2023)
- [Understanding Figurative Meaning through Explainable Visual Entailment](https://mlsanity.com/paper/2405.01474/) (2024)
- [A Survey of Multimodal Sarcasm Detection](https://mlsanity.com/paper/2410.18882/) (2024)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)
- [Can MLLMs Understand the Deep Implication Behind Chinese Images?](https://mlsanity.com/paper/2410.13854/) (2024)
- [What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases](https://mlsanity.com/paper/2404.02415/) (2024)
- [Can Language Models Laugh at YouTube Short-form Videos?](https://mlsanity.com/paper/2310.14159/) (2023)
- [Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding](https://mlsanity.com/paper/2403.11311/) (2024)
- [A Survey in Automatic Irony Processing: Linguistic, Cognitive, and Multi-X Perspectives](https://mlsanity.com/paper/2209.04712/) (2022)
- [Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue](https://mlsanity.com/paper/2402.03658/) (2024)
- [PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension](https://mlsanity.com/paper/2412.11906/) (2024)
- [Let Androids Dream of Electric Sheep: A Human-Inspired Image Implication Understanding and Reasoning Framework](https://mlsanity.com/paper/2505.17019/) (2025)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?](https://mlsanity.com/paper/2502.13925/) (2025)
- [Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models](https://mlsanity.com/paper/2503.12149/) (2025)
- [VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features](https://mlsanity.com/paper/2408.10246/) (2024)
- [Target-Augmented Shared Fusion-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2502.07391/) (2025)
- [Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2510.11852/) (2025)
- [Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition](https://mlsanity.com/paper/2305.13583/) (2023)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Think out Loud: Emotion Deducing Explanation in Dialogues](https://mlsanity.com/paper/2406.04758/) (2024)
- [CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation](https://mlsanity.com/paper/2512.06814/) (2025)
- [MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues](https://mlsanity.com/paper/2601.20451/) (2026)
- [ProCrit: Self-Elicited Multi-Perspective Reasoning with Critic-Guided Revision for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2605.20867/) (2026)

## Abstract (arXiv)

Sarcasm is a pervading linguistic phenomenon and highly challenging to explain due to its subjectivity, lack of context and deeply-felt opinion. In the multimodal setup, sarcasm is conveyed through the incongruity between the text and visual entities. Although recent approaches deal with sarcasm as a classification problem, it is unclear why an online post is identified as sarcastic. Without proper explanation, end users may not be able to perceive the underlying sense of irony. In this paper, we propose a novel problem -- Multimodal Sarcasm Explanation (MuSE) -- given a multimodal sarcastic post containing an image and a caption, we aim to generate a natural language explanation to reveal the intended sarcasm. To this end, we develop MORE, a new dataset with explanation of 3510 sarcastic multimodal posts. Each explanation is a natural language (English) sentence describing the hidden irony. We benchmark MORE by employing a multimodal Transformer-based architecture. It incorporates a cross-modal attention in the Transformer's encoder which attends to the distinguishing features between the two modalities. Subsequently, a BART-based auto-regressive decoder is used as the generator. Empirical results demonstrate convincing results over various baselines (adopted for MuSE) across five evaluation metrics. We also conduct human evaluation on predictions and obtain Fleiss' Kappa score of 0.4 as a fair agreement among 25 evaluators.