mlsanityData to 14 Oct 2026

A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

arXiv 2405.10579 ↗First posted 17 May 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 6 of its 14 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

6 of 14 references.

References · 6

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  4. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  5. 2021AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Modelscited by 27
  6. 2021Investigating Robustness of Dialog Models to Popular Figurative Language Constructscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 7.

11 papers.

  1. 2024Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Contextcited by 11
  2. 2025Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanationcited by 6
  3. 2026When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalitiescited by 4
  4. 2025NLP Datasets for Idiom and Figurative Language Taskscited by 2
  5. 2026FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Languagecited by 2
  6. 2026A Rising Tide Lifts All Boats: MTQE Rewards for Idioms Improve General Translation Qualitycited by 1
  7. 2025Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contextscited by 0
  8. 2026IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressionscited by 0
  9. 2026Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languagescited by 0
  10. 2026G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignmentcited by 0
  11. 2026Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmarkcited by 0