# Finding Challenging Metaphors that Confuse Pretrained Language Models

arXiv: https://arxiv.org/abs/2401.16012 · HTML: https://mlsanity.com/paper/2401.16012/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 17 of 17)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [On Sampling Strategies for Neural Network-based Collaborative Filtering](https://mlsanity.com/paper/1706.07881/) (2017)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories](https://mlsanity.com/paper/2104.13615/) (2021)
- [FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learning](https://mlsanity.com/paper/2302.04834/) (2023)
- [Metaphor Detection with Effective Context Denoising](https://mlsanity.com/paper/2302.05611/) (2023)
- [Metaphor Detection via Explicit Basic Meanings Modelling](https://mlsanity.com/paper/2305.17268/) (2023)
- ["Love is as Complex as Math": Metaphor Generation System for Social Chatbot](https://mlsanity.com/paper/2001.00733/) (2020)
- [The Secret of Metaphor on Expressing Stronger Emotion](https://mlsanity.com/paper/2301.13042/) (2023)
- [Nominal Metaphor Generation with Multitask Learning](https://mlsanity.com/paper/2206.05195/) (2022)

## Built on by (most cited first, 3 of 3)

- [In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models](https://mlsanity.com/paper/2409.15454/) (2024)
- [Overview of the NLPCC 2024 Shared Task on Chinese Metaphor Generation](https://mlsanity.com/paper/2408.04378/) (2024)
- [Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model](https://mlsanity.com/paper/2408.09177/) (2024)

## Abstract (arXiv)

Metaphors are considered to pose challenges for a wide spectrum of NLP tasks. This gives rise to the area of computational metaphor processing. However, it remains unclear what types of metaphors challenge current state-of-the-art models. In this paper, we test various NLP models on the VUA metaphor dataset and quantify to what extent metaphors affect models' performance on various downstream tasks. Analysis reveals that VUA includes a large number of metaphors that pose little difficulty to downstream tasks. We would like to shift the attention of researchers away from these metaphors to instead focus on challenging metaphors. To identify hard metaphors, we propose an automatic pipeline that identifies metaphors that challenge a particular model. Our analysis demonstrates that our detected hard metaphors contrast significantly with VUA and reduce the accuracy of machine translation by 16%, QA performance by 4%, NLI by 7%, and metaphor identification recall by over 14% for various popular NLP systems.