# MiQA: A Benchmark for Inference on Metaphorical Questions

arXiv: https://arxiv.org/abs/2210.07993 · HTML: https://mlsanity.com/paper/2210.07993/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories](https://mlsanity.com/paper/2104.13615/) (2021)
- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)

## Built on by (most cited first, 9 of 9)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Metaphor Understanding Challenge Dataset for LLMs](https://mlsanity.com/paper/2403.11810/) (2024)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation](https://mlsanity.com/paper/2404.07053/) (2024)
- [KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning](https://mlsanity.com/paper/2505.09825/) (2025)
- [Conceptual Metaphor Theory as a Prompting Paradigm for Large Language Models](https://mlsanity.com/paper/2502.01901/) (2025)
- [Hummus: A Dataset of Humorous Multimodal Metaphor Use](https://mlsanity.com/paper/2504.02983/) (2025)
- [Idiom Understanding as a Tool to Measure the Dialect Gap](https://mlsanity.com/paper/2510.05026/) (2025)
- [Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model](https://mlsanity.com/paper/2408.09177/) (2024)

## Abstract (arXiv)

We propose a benchmark to assess the capability of large language models to reason with conventional metaphors. Our benchmark combines the previously isolated topics of metaphor detection and commonsense reasoning into a single task that requires a model to make inferences by accurately selecting between the literal and metaphorical register. We examine the performance of state-of-the-art pre-trained models on binary-choice tasks and find a large discrepancy between the performance of small and very large models, going from chance to near-human level. We also analyse the largest model in a generative setting and find that although human performance is approached, careful multiple-shot prompting is required.