# A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

arXiv: https://arxiv.org/abs/2405.10579 · HTML: https://mlsanity.com/paper/2405.10579/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 6 of 6)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [FLUTE: Figurative Language Understanding through Textual Explanations](https://mlsanity.com/paper/2205.12404/) (2022)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models](https://mlsanity.com/paper/2109.04413/) (2021)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)

## Built on by (most cited first, 11 of 11)

- [Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context](https://mlsanity.com/paper/2410.16069/) (2024)
- [Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation](https://mlsanity.com/paper/2510.00567/) (2025)
- [When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities](https://mlsanity.com/paper/2604.10787/) (2026)
- [NLP Datasets for Idiom and Figurative Language Tasks](https://mlsanity.com/paper/2511.16345/) (2025)
- [FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language](https://mlsanity.com/paper/2601.20105/) (2026)
- [A Rising Tide Lifts All Boats: MTQE Rewards for Idioms Improve General Translation Quality](https://mlsanity.com/paper/2601.06307/) (2026)
- [Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts](https://mlsanity.com/paper/2504.20051/) (2025)
- [IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions](https://mlsanity.com/paper/2605.22247/) (2026)
- [Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages](https://mlsanity.com/paper/2606.02147/) (2026)
- [G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment](https://mlsanity.com/paper/2606.18989/) (2026)
- [Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark](https://mlsanity.com/paper/2608.04670/) (2026)

## Abstract (arXiv)

In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.