# VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features

arXiv: https://arxiv.org/abs/2408.10246 · HTML: https://mlsanity.com/paper/2408.10246/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 16 of 16)

- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [SA-Net: Shuffle Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2102.00240/) (2021)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [RegNet: Self-Regulated Network for Image Classification](https://mlsanity.com/paper/2101.00590/) (2021)
- [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1610.08815/) (2016)
- [A Multimodal Corpus for Emotion Recognition in Sarcasm](https://mlsanity.com/paper/2206.02119/) (2022)
- [CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://mlsanity.com/paper/1805.06413/) (2018)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [A comparative study between vision transformers and CNNs in digital pathology](https://mlsanity.com/paper/2206.00389/) (2022)
- [Are Word Embedding-based Features Useful for Sarcasm Detection?](https://mlsanity.com/paper/1610.00883/) (2016)
- [Representing Social Media Users for Sarcasm Detection](https://mlsanity.com/paper/1808.08470/) (2018)
- [Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection](https://mlsanity.com/paper/1904.04019/) (2019)

## Built on by (most cited first, 3 of 3)

- [Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detection](https://mlsanity.com/paper/2508.10644/) (2025)
- [MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues](https://mlsanity.com/paper/2601.20451/) (2026)
- [PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention](https://mlsanity.com/paper/2605.02447/) (2026)

## Abstract (arXiv)

Various linguistic and non-linguistic clues, such as excessive emphasis on a word, a shift in the tone of voice, or an awkward expression, frequently convey sarcasm. The computer vision problem of sarcasm recognition in conversation aims to identify hidden sarcastic, criticizing, and metaphorical information embedded in everyday dialogue. Prior, sarcasm recognition has focused mainly on text. Still, it is critical to consider all textual information, audio stream, facial expression, and body position for reliable sarcasm identification. Hence, we propose a novel approach that combines a lightweight depth attention module with a self-regulated ConvNet to concentrate on the most crucial features of visual data and an attentional tokenizer based strategy to extract the most critical context-specific information from the textual data. The following is a list of the key contributions that our experimentation has made in response to performing the task of Multi-modal Sarcasm Recognition: an attentional tokenizer branch to get beneficial features from the glossary content provided by the subtitles; a visual branch for acquiring the most prominent features from the video frames; an utterance-level feature extraction from acoustic content and a multi-headed attention based feature fusion branch to blend features obtained from multiple modalities. Extensive testing on one of the benchmark video datasets, MUSTaRD, yielded an accuracy of 79.86% for speaker dependent and 76.94% for speaker independent configuration demonstrating that our approach is superior to the existing methods. We have also conducted a cross-dataset analysis to test the adaptability of VyAnG-Net with unseen samples of another dataset MUStARD++.