# Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations

arXiv: https://arxiv.org/abs/2105.09984 · HTML: https://mlsanity.com/paper/2105.09984/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/1905.05812/) (2019)
- [CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://mlsanity.com/paper/1805.06413/) (2018)
- [End-to-End Environmental Sound Classification using a 1D Convolutional Neural Network](https://mlsanity.com/paper/1904.08990/) (2019)
- [The Role of Conversation Context for Sarcasm Detection in Online Interactions](https://mlsanity.com/paper/1707.06226/) (2017)
- [A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection](https://mlsanity.com/paper/1805.11869/) (2018)
- [Discovering Emotion and Reasoning its Flip in Multi-Party Conversations using Masked Memory Network and Transformer](https://mlsanity.com/paper/2103.12360/) (2021)
- [Humor Detection in English-Hindi Code-Mixed Social Media Content : Corpus and Baseline System](https://mlsanity.com/paper/1806.05513/) (2018)

## Built on by (most cited first, 30 of 33)

- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://mlsanity.com/paper/2203.06419/) (2022)
- [Speaker and Time-aware Joint Contextual Learning for Dialogue-act Classification in Counselling Conversations](https://mlsanity.com/paper/2111.06647/) (2021)
- [A Multimodal Corpus for Emotion Recognition in Sarcasm](https://mlsanity.com/paper/2206.02119/) (2022)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [CAD -- Contextual Multi-modal Alignment for Dynamic AVQA](https://mlsanity.com/paper/2310.16754/) (2023)
- [Explaining (Sarcastic) Utterances to Enhance Affect Understanding in Multimodal Dialogues](https://mlsanity.com/paper/2211.11049/) (2022)
- [Systematic Literature Review: Computational Approaches for Humour Style Classification](https://mlsanity.com/paper/2402.01759/) (2024)
- [SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)](https://mlsanity.com/paper/2402.18944/) (2024)
- [Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results](https://mlsanity.com/paper/2209.14272/) (2022)
- [NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection](https://mlsanity.com/paper/2306.06885/) (2023)
- [Code-mixed Sentiment and Hate-speech Prediction](https://mlsanity.com/paper/2405.12929/) (2024)
- [FanChuan: A Multilingual and Graph-Structured Benchmark For Parody Detection and Analysis](https://mlsanity.com/paper/2502.16503/) (2025)
- [Spoken in Jest, Detected in Earnest: A Systematic Review of Sarcasm Recognition -- Multimodal Fusion, Challenges, and Future Prospects](https://mlsanity.com/paper/2509.04605/) (2025)
- [Target-Augmented Shared Fusion-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2502.07391/) (2025)
- [A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer](https://mlsanity.com/paper/2204.12753/) (2022)
- [From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues](https://mlsanity.com/paper/2310.13080/) (2023)
- [Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2408.02595/) (2024)
- [Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection](https://mlsanity.com/paper/2412.12761/) (2024)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues](https://mlsanity.com/paper/2401.12995/) (2024)
- [A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection](https://mlsanity.com/paper/2503.11838/) (2025)
- [MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations](https://mlsanity.com/paper/2510.24178/) (2025)
- [SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?](https://mlsanity.com/paper/2602.00327/) (2026)
- [IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings](https://mlsanity.com/paper/2404.04525/) (2024)
- [This Paper Had the Smartest Reviewers -- Flattery Detection Utilising an Audio-Textual Transformer-Based Approach](https://mlsanity.com/paper/2406.17667/) (2024)
- [MasonTigers at SemEval-2024 Task 10: Emotion Discovery and Flip Reasoning in Conversation with Ensemble of Transformers and Prompting](https://mlsanity.com/paper/2407.00581/) (2024)
- [AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations](https://mlsanity.com/paper/2501.11166/) (2025)
- [SaRoHead: Detecting Satire in a Multi-Domain Romanian News Headline Dataset](https://mlsanity.com/paper/2504.07612/) (2025)
- [DEBATE: A Dataset for Disentangling Textual Ambiguity in Mandarin Through Speech](https://mlsanity.com/paper/2506.07502/) (2025)

## Abstract (arXiv)

Sarcasm detection and humor classification are inherently subtle problems, primarily due to their dependence on the contextual and non-verbal information. Furthermore, existing studies in these two topics are usually constrained in non-English languages such as Hindi, due to the unavailability of qualitative annotated datasets. In this work, we make two major contributions considering the above limitations: (1) we develop a Hindi-English code-mixed dataset, MaSaC, for the multi-modal sarcasm detection and humor classification in conversational dialog, which to our knowledge is the first dataset of its kind; (2) we propose MSH-COMICS, a novel attention-rich neural architecture for the utterance classification. We learn efficient utterance representation utilizing a hierarchical attention mechanism that attends to a small portion of the input sentence at a time. Further, we incorporate dialog-level contextual attention mechanism to leverage the dialog history for the multi-modal classification. We perform extensive experiments for both the tasks by varying multi-modal inputs and various submodules of MSH-COMICS. We also conduct comparative analysis against existing approaches. We observe that MSH-COMICS attains superior performance over the existing models by > 1 F1-score point for the sarcasm detection and 10 F1-score points in humor classification. We diagnose our model and perform thorough analysis of the results to understand the superiority and pitfalls.