# "I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

arXiv: https://arxiv.org/abs/2604.05930 · HTML: https://mlsanity.com/paper/2604.05930/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 21 of 21)

- [Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels](https://mlsanity.com/paper/2310.14122/) (2023)
- [IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents](https://mlsanity.com/paper/2508.15310/) (2025)
- [ExPUNations: Augmenting Puns with Keywords and Explanations](https://mlsanity.com/paper/2210.13513/) (2022)
- ["A good pun is its own reword": Can Large Language Models Understand Puns?](https://mlsanity.com/paper/2404.13599/) (2024)
- [MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge](https://mlsanity.com/paper/2507.21183/) (2025)
- [CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor](https://mlsanity.com/paper/2509.09703/) (2025)
- [Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions](https://mlsanity.com/paper/2405.19088/) (2024)
- [Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understanding](https://mlsanity.com/paper/2406.10318/) (2024)
- [Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training](https://mlsanity.com/paper/2404.13349/) (2024)
- [Heterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezing](https://mlsanity.com/paper/2408.09101/) (2024)
- [Fingerprint Vector: Enabling Scalable and Efficient Model Fingerprint Transfer via Vector Addition](https://mlsanity.com/paper/2409.08846/) (2024)
- [Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?](https://mlsanity.com/paper/2512.21871/) (2025)
- [Joint Detection and Location of English Puns](https://mlsanity.com/paper/1909.00175/) (2019)
- ["The Boating Store Had Its Best Sail Ever": Pronunciation-attentive Contextualized Pun Recognition](https://mlsanity.com/paper/2004.14457/) (2020)
- [AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing](https://mlsanity.com/paper/2601.11007/) (2026)
- [Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!](https://mlsanity.com/paper/2410.01023/) (2024)
- [Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning](https://mlsanity.com/paper/2512.00902/) (2025)
- [TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings](https://mlsanity.com/paper/2603.04772/) (2026)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [Memory-Efficient Federated Fine-Tuning of Large Language Models via Layer Pruning](https://mlsanity.com/paper/2508.17209/) (2025)
- [Pun Unintended: LLMs and the Illusion of Humor Understanding](https://mlsanity.com/paper/2509.12158/) (2025)

## Built on by (most cited first, 2 of 2)

- [Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation](https://mlsanity.com/paper/2604.10071/) (2026)
- [CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test](https://mlsanity.com/paper/2610.18569/) (2026)

## Abstract (arXiv)

Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.