# Hummus: A Dataset of Humorous Multimodal Metaphor Use

arXiv: https://arxiv.org/abs/2504.02983 · HTML: https://mlsanity.com/paper/2504.02983/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 67 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [Metaphor Understanding Challenge Dataset for LLMs](https://mlsanity.com/paper/2403.11810/) (2024)
- [Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest](https://mlsanity.com/paper/1506.08126/) (2015)
- [Metaphor Detection with Effective Context Denoising](https://mlsanity.com/paper/2302.05611/) (2023)
- [MiQA: A Benchmark for Inference on Metaphorical Questions](https://mlsanity.com/paper/2210.07993/) (2022)
- [Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos](https://mlsanity.com/paper/2301.01134/) (2022)
- [A framework for annotating and modelling intentions behind metaphor use](https://mlsanity.com/paper/2407.03952/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Computational Approach to Visual Metonymy](https://mlsanity.com/paper/2601.17706/) (2026)
- [When Jokes Cross the Line: Analyzing Regular Humor and Dark Humor in YouTube Shorts](https://mlsanity.com/paper/2606.00046/) (2026)

## Abstract (arXiv)

Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.