# Testing the Ability of Language Models to Interpret Figurative Language

arXiv: https://arxiv.org/abs/2204.12632 · HTML: https://mlsanity.com/paper/2204.12632/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)
- [Neural Metaphor Detection in Context](https://mlsanity.com/paper/1808.09653/) (2018)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [Figurative Language in Recognizing Textual Entailment](https://mlsanity.com/paper/2106.01195/) (2021)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)

## Built on by (most cited first, 30 of 67)

- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Stable LM 2 1.6B Technical Report](https://mlsanity.com/paper/2402.17834/) (2024)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [Metaphor Understanding Challenge Dataset for LLMs](https://mlsanity.com/paper/2403.11810/) (2024)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs](https://mlsanity.com/paper/2505.18152/) (2025)
- [Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors](https://mlsanity.com/paper/2308.01497/) (2023)
- [A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models](https://mlsanity.com/paper/2405.10579/) (2024)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [MiQA: A Benchmark for Inference on Metaphorical Questions](https://mlsanity.com/paper/2210.07993/) (2022)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Non-literal Understanding of Number Words by Language Models](https://mlsanity.com/paper/2502.06204/) (2025)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games](https://mlsanity.com/paper/2505.18218/) (2025)
- [A Survey in Automatic Irony Processing: Linguistic, Cognitive, and Multi-X Perspectives](https://mlsanity.com/paper/2209.04712/) (2022)
- [Connecting the Dots in News Analysis: Bridging the Cross-Disciplinary Disparities in Media Bias and Framing](https://mlsanity.com/paper/2309.08069/) (2023)
- [Toward Informal Language Processing: Knowledge of Slang in Large Language Models](https://mlsanity.com/paper/2404.02323/) (2024)
- [Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation](https://mlsanity.com/paper/2510.00567/) (2025)
- [Leveraging a New Spanish Corpus for Multilingual and Crosslingual Metaphor Detection](https://mlsanity.com/paper/2210.10358/) (2022)
- [EUREKA: EUphemism Recognition Enhanced through Knn-based methods and Augmentation](https://mlsanity.com/paper/2210.12846/) (2022)
- [Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation](https://mlsanity.com/paper/2404.07053/) (2024)
- [Semantic Change Characterization with LLMs using Rhetorics](https://mlsanity.com/paper/2407.16624/) (2024)
- [When People are Floods: Analyzing Dehumanizing Metaphors in Immigration Discourse with Large Language Models](https://mlsanity.com/paper/2502.13246/) (2025)

## Abstract (arXiv)

Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.