# To What Extent Do Large Language Models Understand Bangla Idioms?

arXiv: https://arxiv.org/abs/2609.03410 · HTML: https://mlsanity.com/paper/2609.03410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [BanglaParaphrase: A High-Quality Bangla Paraphrase Dataset](https://mlsanity.com/paper/2210.05109/) (2022)
- [Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection](https://mlsanity.com/paper/2405.09279/) (2024)
- [Simple or Complex? Learning to Predict Readability of Bengali Texts](https://mlsanity.com/paper/2012.07701/) (2020)
- [When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities](https://mlsanity.com/paper/2604.10787/) (2026)
- [Sample Efficient Approaches for Idiomaticity Detection](https://mlsanity.com/paper/2205.11306/) (2022)
- [Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space](https://mlsanity.com/paper/2304.14333/) (2023)
- [Idiom Understanding as a Tool to Measure the Dialect Gap](https://mlsanity.com/paper/2510.05026/) (2025)
- [When Words Don't Mean What They Say: Figurative Understanding in Bengali Idioms](https://mlsanity.com/paper/2602.12921/) (2026)

## Abstract (arXiv)

Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low-resource languages. In this paper, we present the first large-scale benchmark dataset of Bangla idioms, complemented by a synthetic multiple-choice question (MCQ) dataset for idiom meaning identification. We conduct a comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies. Our results reveal substantial variability in model performance, with no single LLM consistently outperforming others across all tasks. Notably, Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification. We believe that our datasets and analyses will provide valuable resources to guide future research in improving LLM comprehension of idiomatic expressions, particularly in Bangla and other low-resource languages.