# AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2109.04413 · HTML: https://mlsanity.com/paper/2109.04413/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Assessing Phrasal Representation and Composition in Transformers](https://mlsanity.com/paper/2010.03763/) (2020)
- [Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition](https://mlsanity.com/paper/1902.10618/) (2019)
- [Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts](https://mlsanity.com/paper/1904.01617/) (2019)
- [SemEval-2013 Task 4: Free Paraphrases of Noun Compounds](https://mlsanity.com/paper/1911.10421/) (2019)
- [Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings](https://mlsanity.com/paper/1603.06067/) (2016)
- [Resolving Lexical Ambiguity in Tensor Regression Models of Meaning](https://mlsanity.com/paper/1408.6181/) (2014)

## Built on by (most cited first, 27 of 27)

- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection](https://mlsanity.com/paper/2405.09279/) (2024)
- [A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models](https://mlsanity.com/paper/2405.10579/) (2024)
- [Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context](https://mlsanity.com/paper/2410.16069/) (2024)
- [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://mlsanity.com/paper/2210.04545/) (2022)
- [Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss](https://mlsanity.com/paper/2406.15175/) (2024)
- [drsphelps at SemEval-2022 Task 2: Learning idiom representations using BERTRAM](https://mlsanity.com/paper/2204.02821/) (2022)
- [Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions](https://mlsanity.com/paper/2207.03679/) (2022)
- [HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection](https://mlsanity.com/paper/2204.06145/) (2022)
- [Sample Efficient Approaches for Idiomaticity Detection](https://mlsanity.com/paper/2205.11306/) (2022)
- [SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation](https://mlsanity.com/paper/2503.15358/) (2025)
- [Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5](https://mlsanity.com/paper/2210.17301/) (2022)
- [A Survey of Idiom Datasets for Psycholinguistic and Computational Research](https://mlsanity.com/paper/2508.11828/) (2025)
- [JARVix at SemEval-2022 Task 2: It Takes One to Know One? Idiomaticity Detection using Zero and One-Shot Learning](https://mlsanity.com/paper/2202.02394/) (2022)
- [kpfriends at SemEval-2022 Task 2: NEAMER -- Named Entity Augmented Multi-word Expression Recognizer](https://mlsanity.com/paper/2204.08102/) (2022)
- [UAlberta at SemEval 2022 Task 2: Leveraging Glosses and Translations for Multilingual Idiomaticity Detection](https://mlsanity.com/paper/2205.14084/) (2022)
- [Generating Continuations in Multilingual Idiomatic Contexts](https://mlsanity.com/paper/2310.20195/) (2023)
- [IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions](https://mlsanity.com/paper/2312.06053/) (2023)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)
- [A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding](https://mlsanity.com/paper/2601.08645/) (2026)
- [SinFoS: A Parallel Dataset for Translating Sinhala Figures of Speech](https://mlsanity.com/paper/2602.09866/) (2026)
- [HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Models](https://mlsanity.com/paper/2205.13708/) (2022)
- [OCHADAI at SemEval-2022 Task 2: Adversarial Training for Multilingual Idiomaticity Detection](https://mlsanity.com/paper/2206.03025/) (2022)
- [HYU at SemEval-2022 Task 2: Effective Idiomaticity Detection with Consideration at Different Levels of Contextualization](https://mlsanity.com/paper/2206.11854/) (2022)
- [Assessing Linguistic Generalisation in Language Models: A Dataset for Brazilian Portuguese](https://mlsanity.com/paper/2305.14070/) (2023)
- [Unsupervised Paraphrasing of Multiword Expressions](https://mlsanity.com/paper/2306.01443/) (2023)
- [Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages](https://mlsanity.com/paper/2606.02147/) (2026)

## Abstract (arXiv)

Despite their success in a variety of NLP tasks, pre-trained language models, due to their heavy reliance on compositionality, fail in effectively capturing the meanings of multiword expressions (MWEs), especially idioms. Therefore, datasets and methods to improve the representation of MWEs are urgently needed. Existing datasets are limited to providing the degree of idiomaticity of expressions along with the literal and, where applicable, (a single) non-literal interpretation of MWEs. This work presents a novel dataset of naturally occurring sentences containing MWEs manually classified into a fine-grained set of meanings, spanning both English and Portuguese. We use this dataset in two tasks designed to test i) a language model's ability to detect idiom usage, and ii) the effectiveness of a language model in generating representations of sentences containing idioms. Our experiments demonstrate that, on the task of detecting idiomatic usage, these models perform reasonably well in the one-shot and few-shot scenarios, but that there is significant scope for improvement in the zero-shot scenario. On the task of representing idiomaticity, we find that pre-training is not always effective, while fine-tuning could provide a sample efficient method of learning representations of sentences containing MWEs.