# Construction Artifacts in Metaphor Identification Datasets

arXiv: https://arxiv.org/abs/2311.00790 · HTML: https://mlsanity.com/paper/2311.00790/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding](https://mlsanity.com/paper/2103.06779/) (2021)
- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Neural Metaphor Detection in Context](https://mlsanity.com/paper/1808.09653/) (2018)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://mlsanity.com/paper/1603.06076/) (2016)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor](https://mlsanity.com/paper/1905.09866/) (2019)
- [Metaphor Detection with Effective Context Denoising](https://mlsanity.com/paper/2302.05611/) (2023)
- [Probing Pre-Trained Language Models for Disease Knowledge](https://mlsanity.com/paper/2106.07285/) (2021)
- [Grasping the Finer Point: A Supervised Similarity Network for Metaphor Detection](https://mlsanity.com/paper/1709.00575/) (2017)

## Built on by (most cited first, 5 of 5)

- [Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection](https://mlsanity.com/paper/2405.09279/) (2024)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation](https://mlsanity.com/paper/2503.15358/) (2025)
- [MetaphorShare: A Dynamic Collaborative Repository of Open Metaphor Datasets](https://mlsanity.com/paper/2411.18260/) (2024)
- [A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding](https://mlsanity.com/paper/2601.08645/) (2026)

## Abstract (arXiv)

Metaphor identification aims at understanding whether a given expression is used figuratively in context. However, in this paper we show how existing metaphor identification datasets can be gamed by fully ignoring the potential metaphorical expression or the context in which it occurs. We test this hypothesis in a variety of datasets and settings, and show that metaphor identification systems based on language models without complete information can be competitive with those using the full context. This is due to the construction procedures to build such datasets, which introduce unwanted biases for positive and negative classes. Finally, we test the same hypothesis on datasets that are carefully sampled from natural corpora and where this bias is not present, making these datasets more challenging and reliable.