mlsanityData to 14 Oct 2026

Metaphor Understanding Challenge Dataset for LLMs

arXiv 2403.11810 ↗First posted 18 Mar 2024cs.CLcited by 30

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 52 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

16 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  6. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  7. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  8. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  9. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  10. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  11. 2021MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theoriescited by 46
  12. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
3 more
  1. 2023FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learningcited by 18
  2. 2022MiQA: A Benchmark for Inference on Metaphorical Questionscited by 9
  3. 2022Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 12.

30 papers.

Sort:

By area:

  1. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  2. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 17
  3. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  4. 2025Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphorscited by 10
  5. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  6. 2025Metaphor and Large Language Models: When Surface Features Matter More than Deep Understandingcited by 8
  7. 2025CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Gamescited by 7
  8. 2024Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluationcited by 5
  9. 2025Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processingcited by 5
  10. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  11. 2025KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoningcited by 4
  12. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosiscited by 3
18 more
  1. 2025Hummus: A Dataset of Humorous Multimodal Metaphor Usecited by 2
  2. 2026Not all ANIMALs are equal: metaphorical framing through source domains and semantic framescited by 2
  3. 2024MetaphorShare: A Dynamic Collaborative Repository of Open Metaphor Datasetscited by 1
  4. 2025Breaking Bad: Norms for Valence, Arousal, and Dominance for over 10k English Multiword Expressionscited by 1
  5. 2026Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasetscited by 1
  6. 2024Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Modelcited by 0
  7. 2025The Medical Metaphors Corpus (MCC)cited by 0
  8. 2025Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Mediacited by 0
  9. 2025The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mappingcited by 0
  10. 2026From Trial by Fire To Sleep Like a Baby: A Lexicon of Anxiety Associations for 20k English Multiword Expressionscited by 0
  11. 2026Exploring Concreteness Through a Figurative Lenscited by 0
  12. 2026DRInQ: Evaluating Conversational Implicature with Controlled Context Variationcited by 0
  13. 2026Playing with Words, Improving with Rewards: Training Language Models for Creative Associationcited by 0
  14. 2026As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Languagecited by 0
  15. 2026Towards High-Level Semantic Intelligencecited by 0
  16. 2026On the Diversity of Analogy Making in Large Language Modelscited by 0
  17. 2026Simile Understanding in Text-to-Image Models: An Evaluation Frameworkcited by 0
  18. 2026Discovering Conceptual Metaphors Across Topics and Media Typescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory
  2. 2022Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models
  3. 2024A Dual-Perspective Metaphor Detection Framework Using Large Language Models
  4. 2025Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification
  5. 2025Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning
  6. 2025Enhancing multimodal analogical reasoning with Logic Augmented Generation
  7. 2026Metaphors are a Source of Cross-Domain Misalignment of Large Reasoning Models
  8. 2026M3R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding