# Metaphor Understanding Challenge Dataset for LLMs

arXiv: https://arxiv.org/abs/2403.11810 · HTML: https://mlsanity.com/paper/2403.11810/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 52 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories](https://mlsanity.com/paper/2104.13615/) (2021)
- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)
- [FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learning](https://mlsanity.com/paper/2302.04834/) (2023)
- [MiQA: A Benchmark for Inference on Metaphorical Questions](https://mlsanity.com/paper/2210.07993/) (2022)
- [Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Models](https://mlsanity.com/paper/2203.16474/) (2022)

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Evaluation of Cognitive Biases in LLMs](https://mlsanity.com/paper/2410.15413/) (2024)
- [AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models](https://mlsanity.com/paper/2505.17312/) (2025)
- [The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding](https://mlsanity.com/paper/2502.08946/) (2025)
- [Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors](https://mlsanity.com/paper/2506.06987/) (2025)
- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games](https://mlsanity.com/paper/2505.18218/) (2025)
- [Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation](https://mlsanity.com/paper/2412.15375/) (2024)
- [Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing](https://mlsanity.com/paper/2510.04120/) (2025)
- [From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis](https://mlsanity.com/paper/2501.06101/) (2025)
- [KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning](https://mlsanity.com/paper/2505.09825/) (2025)
- [Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis](https://mlsanity.com/paper/2604.02359/) (2026)
- [Hummus: A Dataset of Humorous Multimodal Metaphor Use](https://mlsanity.com/paper/2504.02983/) (2025)
- [Not all ANIMALs are equal: metaphorical framing through source domains and semantic frames](https://mlsanity.com/paper/2604.20454/) (2026)
- [MetaphorShare: A Dynamic Collaborative Repository of Open Metaphor Datasets](https://mlsanity.com/paper/2411.18260/) (2024)
- [Breaking Bad: Norms for Valence, Arousal, and Dominance for over 10k English Multiword Expressions](https://mlsanity.com/paper/2511.19816/) (2025)
- [Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets](https://mlsanity.com/paper/2601.02015/) (2026)
- [Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model](https://mlsanity.com/paper/2408.09177/) (2024)
- [The Medical Metaphors Corpus (MCC)](https://mlsanity.com/paper/2508.07993/) (2025)
- [Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Media](https://mlsanity.com/paper/2510.08365/) (2025)
- [The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping](https://mlsanity.com/paper/2510.08482/) (2025)
- [From Trial by Fire To Sleep Like a Baby: A Lexicon of Anxiety Associations for 20k English Multiword Expressions](https://mlsanity.com/paper/2602.18692/) (2026)
- [Exploring Concreteness Through a Figurative Lens](https://mlsanity.com/paper/2604.18296/) (2026)
- [DRInQ: Evaluating Conversational Implicature with Controlled Context Variation](https://mlsanity.com/paper/2605.24267/) (2026)
- [Playing with Words, Improving with Rewards: Training Language Models for Creative Association](https://mlsanity.com/paper/2605.27832/) (2026)
- [As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language](https://mlsanity.com/paper/2606.18922/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [On the Diversity of Analogy Making in Large Language Models](https://mlsanity.com/paper/2608.03233/) (2026)
- [Simile Understanding in Text-to-Image Models: An Evaluation Framework](https://mlsanity.com/paper/2608.04750/) (2026)
- [Discovering Conceptual Metaphors Across Topics and Media Types](https://mlsanity.com/paper/2608.06652/) (2026)

## Abstract (arXiv)

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.