# Memorization or Reasoning? Exploring the Idiom Understanding of LLMs

arXiv: https://arxiv.org/abs/2505.16216 · HTML: https://mlsanity.com/paper/2505.16216/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [ChID: A Large-scale Chinese IDiom Dataset for Cloze Test](https://mlsanity.com/paper/1906.01265/) (2019)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Multiple-Choice Questions are Efficient and Robust LLM Evaluators](https://mlsanity.com/paper/2405.11966/) (2024)
- [EPIE Dataset: A Corpus For Possible Idiomatic Expressions](https://mlsanity.com/paper/2006.09479/) (2020)
- [Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models](https://mlsanity.com/paper/2308.13961/) (2023)
- [LIDIOMS: A Multilingual Linked Idioms Data Set](https://mlsanity.com/paper/1802.08148/) (2018)
- [Potential Idiomatic Expression (PIE)-English: Corpus for Classes of Idioms](https://mlsanity.com/paper/2105.03280/) (2021)
- [From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?](https://mlsanity.com/paper/2305.10568/) (2023)
- [PETCI: A Parallel English Translation Dataset of Chinese Idioms](https://mlsanity.com/paper/2202.09509/) (2022)
- [Comparative Study of Multilingual Idioms and Similes in Large Language Models](https://mlsanity.com/paper/2410.16461/) (2024)
- [Systematicity in GPT-3's Interpretation of Novel English Noun Compounds](https://mlsanity.com/paper/2210.09492/) (2022)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context](https://mlsanity.com/paper/2412.07251/) (2024)
- [That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?](https://mlsanity.com/paper/2310.14610/) (2023)

## Built on by (most cited first, 7 of 7)

- [LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories](https://mlsanity.com/paper/2509.20909/) (2025)
- [Idiom Understanding as a Tool to Measure the Dialect Gap](https://mlsanity.com/paper/2510.05026/) (2025)
- [A Rising Tide Lifts All Boats: MTQE Rewards for Idioms Improve General Translation Quality](https://mlsanity.com/paper/2601.06307/) (2026)
- [MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs](https://mlsanity.com/paper/2601.22050/) (2026)
- [IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions](https://mlsanity.com/paper/2605.22247/) (2026)
- [Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages](https://mlsanity.com/paper/2606.02147/) (2026)
- [Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark](https://mlsanity.com/paper/2608.04670/) (2026)

## Abstract (arXiv)

Idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. While recent studies have leveraged large language models (LLMs) to handle idioms across various tasks, e.g., idiom-containing sentence generation and idiomatic machine translation, little is known about the underlying mechanisms of idiom processing in LLMs, particularly in multilingual settings. To this end, we introduce MIDAS, a new large-scale dataset of idioms in six languages, each paired with its corresponding meaning. Leveraging this resource, we conduct a comprehensive evaluation of LLMs' idiom processing ability, identifying key factors that influence their performance. Our findings suggest that LLMs rely not only on memorization, but also adopt a hybrid approach that integrates contextual cues and reasoning, especially when processing compositional idioms. This implies that idiom understanding in LLMs emerges from an interplay between internal knowledge retrieval and reasoning-based inference.