mlsanityData to 14 Oct 2026

Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection

arXiv 2405.09279 ↗First posted 15 May 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite the recent ubiquity of large language models and their high zero-shot prompted performance across a wide range of tasks, it is still not known how well they perform on tasks which require processing of potentially idiomatic language. In particular, how well do such models perform in comparison to encoder-only models fine-tuned specifically for idiomaticity tasks? In this work, we attempt to answer this question by looking at the performance of a range of LLMs (both local and software-as-a-service models) on three idiomaticity datasets: SemEval 2022 Task 2a, FLUTE, and MAGPIE. Overall, we find that whilst these models do give competitive performance, they do not match the results of fine-tuned task-specific models, even at the largest scales (e.g. for GPT-4). Nevertheless, we do see consistent performance improvements across model scale. Additionally, we investigate prompting approaches to improve performance, and discuss the practicalities of using LLMs for these tasks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

18 of 33 references.

References · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  9. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  10. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  11. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  12. 2022SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embeddingcited by 36
6 more
  1. 2021AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Modelscited by 27
  2. 2021Idiomatic Expression Identification using Semantic Compatibilitycited by 19
  3. 2022Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTEcited by 8
  4. 2023Construction Artifacts in Metaphor Identification Datasetscited by 5
  5. 2022HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detectioncited by 3
  6. 2022Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2024Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Contextcited by 11
  2. 2025SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representationcited by 3
  3. 2025UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representationcited by 2
  4. 2025NLP Datasets for Idiom and Figurative Language Taskscited by 2
  5. 2025From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errorscited by 1
  6. 2026A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understandingcited by 1
  7. 2025Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contextscited by 0
  8. 2026PolyFrame at MWE-2026 AdMIRe 2: When Words Are Not Enough: Multimodal Idiom Disambiguationcited by 0
  9. 2026IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressionscited by 0
  10. 2026VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLPcited by 0
  11. 2026To What Extent Do Large Language Models Understand Bangla Idioms?cited by 0