# Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection

arXiv: https://arxiv.org/abs/2510.11852 · HTML: https://mlsanity.com/paper/2510.11852/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Delta Denoising Score](https://mlsanity.com/paper/2304.07090/) (2023)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse](https://mlsanity.com/paper/2401.01523/) (2024)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning](https://mlsanity.com/paper/2302.04858/) (2023)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [MUDES: Multilingual Detection of Offensive Spans](https://mlsanity.com/paper/2102.09665/) (2021)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [Automatic Sarcasm Detection: A Survey](https://mlsanity.com/paper/1602.03426/) (2016)
- [A Survey of Multimodal Sarcasm Detection](https://mlsanity.com/paper/2410.18882/) (2024)
- [MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks](https://mlsanity.com/paper/2310.09036/) (2023)
- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Sarcasm Detection: A Comparative Study](https://mlsanity.com/paper/2107.02276/) (2021)
- [BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English](https://mlsanity.com/paper/2412.04726/) (2024)

## Built on by (most cited first, 3 of 3)

- [Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models](https://mlsanity.com/paper/2512.14926/) (2025)
- [MUNIChus: Multilingual News Image Captioning Benchmark](https://mlsanity.com/paper/2603.10613/) (2026)
- [PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models](https://mlsanity.com/paper/2608.09772/) (2026)

## Abstract (arXiv)

Recent advances in open-source vision-language models (VLMs) offer new opportunities for understanding complex and subjective multimodal phenomena such as sarcasm. In this work, we evaluate seven state-of-the-art VLMs - BLIP2, InstructBLIP, OpenFlamingo, LLaVA, PaliGemma, Gemma3, and Qwen-VL - on their ability to detect multimodal sarcasm using zero-, one-, and few-shot prompting. Furthermore, we evaluate the models' capabilities in generating explanations to sarcastic instances. We evaluate the capabilities of VLMs on three benchmark sarcasm datasets (Muse, MMSD2.0, and SarcNet). Our primary objectives are twofold: (1) to quantify each model's performance in detecting sarcastic image-caption pairs, and (2) to assess their ability to generate human-quality explanations that highlight the visual-textual incongruities driving sarcasm. Our results indicate that, while current models achieve moderate success in binary sarcasm detection, they are still not able to generate high-quality explanations without task-specific finetuning.