# Text Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor

arXiv: https://arxiv.org/abs/2412.05315 · HTML: https://mlsanity.com/paper/2412.05315/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- ["A good pun is its own reword": Can Large Language Models Understand Puns?](https://mlsanity.com/paper/2404.13599/) (2024)
- [Context-Situated Pun Generation](https://mlsanity.com/paper/2210.13522/) (2022)

## Built on by (most cited first, 8 of 8)

- [Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs](https://mlsanity.com/paper/2502.20356/) (2025)
- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Who's Laughing Now? An Overview of Computational Humour Generation and Explanation](https://mlsanity.com/paper/2509.21175/) (2025)
- [On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation](https://mlsanity.com/paper/2602.06423/) (2026)
- [From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme](https://mlsanity.com/paper/2512.24555/) (2025)
- [Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor](https://mlsanity.com/paper/2603.17759/) (2026)
- [Re-defining Humor Data Objects for AI Humor Research](https://mlsanity.com/paper/2605.25171/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) have demonstrated impressive natural language understanding capabilities across various text-based tasks, understanding humor has remained a persistent challenge. Humor is frequently multimodal, relying on phonetic ambiguity, rhythm and timing to convey meaning. In this study, we explore a simple multimodal prompting approach to humor understanding and explanation. We present an LLM with both the text and the spoken form of a joke, generated using an off-the-shelf text-to-speech (TTS) system. Using multimodal cues improves the explanations of humor compared to textual prompts across all tested datasets.