# Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

arXiv: https://arxiv.org/abs/1906.01815 · HTML: https://mlsanity.com/paper/1906.01815/

## Key facts

- Cited by 174 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2019.

## Built on by (most cited first, 30 of 174)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations](https://mlsanity.com/paper/2402.12786/) (2024)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [Distilling an End-to-End Voice Assistant Without Instruction Training Data](https://mlsanity.com/paper/2410.02678/) (2024)
- [Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection](https://mlsanity.com/paper/2110.10949/) (2021)
- [Modular and Parameter-Efficient Multimodal Fusion with Prompting](https://mlsanity.com/paper/2203.08055/) (2022)
- [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023)
- [Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations](https://mlsanity.com/paper/2105.09984/) (2021)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://mlsanity.com/paper/2203.06419/) (2022)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models](https://mlsanity.com/paper/2502.04424/) (2025)
- [Understanding the Emergence of Multimodal Representation Alignment](https://mlsanity.com/paper/2502.16282/) (2025)
- [A Multimodal Corpus for Emotion Recognition in Sarcasm](https://mlsanity.com/paper/2206.02119/) (2022)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue](https://mlsanity.com/paper/2312.15316/) (2023)
- [Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications](https://mlsanity.com/paper/2306.04539/) (2023)
- [MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts](https://mlsanity.com/paper/2311.09580/) (2023)
- [A Survey of Multimodal Sarcasm Detection](https://mlsanity.com/paper/2410.18882/) (2024)
- [See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization](https://mlsanity.com/paper/2105.09601/) (2021)
- [Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning](https://mlsanity.com/paper/2404.09403/) (2024)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)

## Abstract (arXiv)

Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. In this paper, we argue that incorporating multimodal cues can improve the automatic classification of sarcasm. As a first step towards enabling the development of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUStARD), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical utterances in the dialogue, which provides additional information on the scenario where the utterance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9% in F-score when compared to the use of individual modalities. The full dataset is publicly available for use at https://github.com/soujanyaporia/MUStARD