# Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2503.12149 · HTML: https://mlsanity.com/paper/2503.12149/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 83 references that are in the map.
- First posted on arXiv 15 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 38)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://mlsanity.com/paper/2203.06419/) (2022)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs](https://mlsanity.com/paper/2503.10084/) (2025)
- [A Multimodal Corpus for Emotion Recognition in Sarcasm](https://mlsanity.com/paper/2206.02119/) (2022)
- [Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement](https://mlsanity.com/paper/2210.03501/) (2022)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Intervention](https://mlsanity.com/paper/2406.05344/) (2024)
- [SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding](https://mlsanity.com/paper/2408.11319/) (2024)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2306.16650/) (2023)
- [A Survey of Multimodal Sarcasm Detection](https://mlsanity.com/paper/2410.18882/) (2024)
- [Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection](https://mlsanity.com/paper/2412.04509/) (2024)
- [Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?](https://mlsanity.com/paper/2407.12725/) (2024)
- [Debiasing Multimodal Sarcasm Detection with Contrastive Learning](https://mlsanity.com/paper/2312.10493/) (2023)
- [M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension](https://mlsanity.com/paper/2407.01131/) (2024)

## Built on by (most cited first, 4 of 4)

- [RefAM: Attention Magnets for Zero-Shot Referral Segmentation](https://mlsanity.com/paper/2509.22650/) (2025)
- [Training-Free Multimodal Deepfake Detection via Graph Reasoning](https://mlsanity.com/paper/2509.21774/) (2025)
- [Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning](https://mlsanity.com/paper/2604.26250/) (2026)
- [Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2608.19942/) (2026)

## Abstract (arXiv)

With the advent of large vision-language models (LVLMs) demonstrating increasingly human-like abilities, a pivotal question emerges: do different LVLMs interpret multimodal sarcasm differently, and can a single model grasp sarcasm from multiple perspectives like humans? To explore this, we introduce an analytical framework using systematically designed prompts on existing multimodal sarcasm datasets. Evaluating 12 state-of-the-art LVLMs over 2,409 samples, we examine interpretive variations within and across models, focusing on confidence levels, alignment with dataset labels, and recognition of ambiguous "neutral" cases. We further validate our findings on a diverse 100-sample mini-benchmark, incorporating multiple datasets, expanded prompt variants, and representative commercial LVLMs. Our findings reveal notable discrepancies -- across LVLMs and within the same model under varied prompts. While classification-oriented prompts yield higher internal consistency, models diverge markedly when tasked with interpretive reasoning. These results challenge binary labeling paradigms by highlighting sarcasm's subjectivity. We advocate moving beyond rigid annotation schemes toward multi-perspective, uncertainty-aware modeling, offering deeper insights into multimodal sarcasm comprehension. Our code and data are available at: https://github.com/CoderChen01/LVLMSarcasmAnalysis