# Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

arXiv: https://arxiv.org/abs/2203.14139 · HTML: https://mlsanity.com/paper/2203.14139/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories](https://mlsanity.com/paper/2104.13615/) (2021)
- [What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties](https://mlsanity.com/paper/2009.12862/) (2020)
- [Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond](https://mlsanity.com/paper/2005.06249/) (2020)
- [Quantifying the Contextualization of Word Representations with Semantic Class Probing](https://mlsanity.com/paper/2004.12198/) (2020)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings](https://mlsanity.com/paper/1906.03608/) (2019)
- [Probing Multilingual Sentence Representations With X-Probe](https://mlsanity.com/paper/1906.05061/) (2019)
- [Intrinsic Subspace Evaluation of Word Embedding Representations](https://mlsanity.com/paper/1606.07902/) (2016)
- [LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations](https://mlsanity.com/paper/1907.11438/) (2019)

## Built on by (most cited first, 30 of 30)

- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [MemeCap: A Dataset for Captioning and Interpreting Memes](https://mlsanity.com/paper/2305.13703/) (2023)
- [Holmes: A Benchmark to Assess the Linguistic Competence of Language Models](https://mlsanity.com/paper/2404.18923/) (2024)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [A Match Made in Heaven: A Multi-task Framework for Hyperbole and Metaphor Detection](https://mlsanity.com/paper/2305.17480/) (2023)
- [MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language](https://mlsanity.com/paper/2406.13698/) (2024)
- [Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors](https://mlsanity.com/paper/2506.06987/) (2025)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [Unveiling the Invisible: Captioning Videos with Metaphors](https://mlsanity.com/paper/2406.04886/) (2024)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization](https://mlsanity.com/paper/2402.01375/) (2024)
- [Toward Informal Language Processing: Knowledge of Slang in Large Language Models](https://mlsanity.com/paper/2404.02323/) (2024)
- [Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation](https://mlsanity.com/paper/2404.07053/) (2024)
- [When People are Floods: Analyzing Dehumanizing Metaphors in Immigration Discourse with Large Language Models](https://mlsanity.com/paper/2502.13246/) (2025)
- [Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing](https://mlsanity.com/paper/2510.04120/) (2025)
- [Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory](https://mlsanity.com/paper/2410.08991/) (2024)
- [Probing with Noise: Unpicking the Warp and Weft of Embeddings](https://mlsanity.com/paper/2210.12206/) (2022)
- [Multilingual Multi-Figurative Language Detection](https://mlsanity.com/paper/2306.00121/) (2023)
- [Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios](https://mlsanity.com/paper/2309.10744/) (2023)
- [Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples](https://mlsanity.com/paper/2306.01206/) (2023)
- [Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets](https://mlsanity.com/paper/2601.02015/) (2026)
- [Systematic word meta-sense extension](https://mlsanity.com/paper/2311.13029/) (2023)
- [Mechanistic Interpretability of Socio-Political Frames in Language Models](https://mlsanity.com/paper/2510.03799/) (2025)
- [Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy](https://mlsanity.com/paper/2605.21391/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Figurative Justice: Detecting metaphors in Hindi judgements with qualitative assessment and transformers](https://mlsanity.com/paper/2608.22446/) (2026)
- [Cross-Dataset Stability of Expert-Informed Skill Prompting and Fine-Tuning for Chinese Metaphor Identification](https://mlsanity.com/paper/2608.25579/) (2026)

## Abstract (arXiv)

Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre-trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross-dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive experiments suggest that contextual representations in PLMs do encode metaphorical knowledge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cognitive and NLP scientists.