# Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

arXiv: https://arxiv.org/abs/1902.10618 · HTML: https://mlsanity.com/paper/1902.10618/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2019.

## Built on by (most cited first, 27 of 27)

- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)
- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models](https://mlsanity.com/paper/2109.04413/) (2021)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking](https://mlsanity.com/paper/2503.00231/) (2025)
- [Scope Ambiguities in Large Language Models](https://mlsanity.com/paper/2404.04332/) (2024)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Generative linguistics contribution to artificial intelligence: Where this contribution lies?](https://mlsanity.com/paper/2410.20221/) (2024)
- [What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysis](https://mlsanity.com/paper/2411.19858/) (2024)
- [A Psycholinguistic Analysis of BERT's Representations of Compounds](https://mlsanity.com/paper/2302.07232/) (2023)
- [Sample Efficient Approaches for Idiomaticity Detection](https://mlsanity.com/paper/2205.11306/) (2022)
- [Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space](https://mlsanity.com/paper/2304.14333/) (2023)
- [Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers](https://mlsanity.com/paper/2205.11456/) (2022)
- [Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Type](https://mlsanity.com/paper/2502.06086/) (2025)
- [UAlberta at SemEval 2022 Task 2: Leveraging Glosses and Translations for Multilingual Idiomaticity Detection](https://mlsanity.com/paper/2205.14084/) (2022)
- [Language Cognition and Language Computation -- Human and Machine Language Understanding](https://mlsanity.com/paper/2301.04788/) (2023)
- [Agentivit`a e telicit`a in GilBERTo: implicazioni cognitive](https://mlsanity.com/paper/2307.02910/) (2023)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)
- [Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders](https://mlsanity.com/paper/2104.05928/) (2021)
- [HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Models](https://mlsanity.com/paper/2205.13708/) (2022)
- [NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?](https://mlsanity.com/paper/2211.04364/) (2022)
- [Compositional Fusion of Signals in Data Embedding](https://mlsanity.com/paper/2311.11085/) (2023)
- [Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax](https://mlsanity.com/paper/2505.06062/) (2025)
- [Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing](https://mlsanity.com/paper/2605.01073/) (2026)

## Abstract (arXiv)

Building meaningful phrase representations is challenging because phrase meanings are not simply the sum of their constituent meanings. Lexical composition can shift the meanings of the constituent words and introduce implicit information. We tested a broad range of textual representations for their capacity to address these issues. We found that as expected, contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information, in which their performance is still far from that of humans. Our evaluation suite, including 5 tasks related to lexical composition effects, can serve future research aiming to improve such representations.