mlsanityData to 14 Oct 2026

Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

arXiv 2410.16069 ↗First posted 21 Oct 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Human processing of idioms relies on understanding the contextual sentences in which idioms occur, as well as language-intrinsic features such as frequency and speaker-intrinsic factors like familiarity. While LLMs have shown high performance on idiomaticity detection tasks, this success may be attributed to reasoning shortcuts in existing datasets. To this end, we construct a novel, controlled contrastive dataset designed to test whether LLMs can effectively use context to disambiguate idiomatic meaning. Additionally, we explore how collocational frequency and sentence probability influence model performance. Our findings reveal that LLMs often fail to resolve idiomaticity when it is required to attend to the surrounding context, and that models perform better on sentences that have higher likelihood. The collocational frequency of expressions also impacts performance. We make our code and dataset publicly available.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 15 of its 58 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

15 of 58 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Large Language Modelscited by 780
  4. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  5. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  6. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  7. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  8. 2018Examining the Tip of the Iceberg: A Data Set for Idiom Translationcited by 34
  9. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  10. 2021AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Modelscited by 27
  11. 2021Idiomatic Expression Identification using Semantic Compatibilitycited by 19
  12. 2023From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?cited by 12
3 more
  1. 2024Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detectioncited by 11
  2. 2024A Hard Nut to Crack: Idiom Detection with Conversational Large Language Modelscited by 11
  3. 2022Systematicity in GPT-3's Interpretation of Novel English Noun Compoundscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representationcited by 3
  2. 2025Tug-of-war between idioms' figurative and literal interpretations in LLMscited by 3
  3. 2025BRoverbs -- Measuring how much LLMs understand Portuguese proverbscited by 3
  4. 2025From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errorscited by 1
  5. 2026A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understandingcited by 1
  6. 2025Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detectioncited by 0
  7. 2026IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressionscited by 0
  8. 2026Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languagescited by 0
  9. 2026G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignmentcited by 0
  10. 2026MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priorscited by 0
  11. 2026It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Themcited by 0