# Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles

arXiv: https://arxiv.org/abs/2512.20324 · HTML: https://mlsanity.com/paper/2512.20324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 15 of 15)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?](https://mlsanity.com/paper/1910.12391/) (2019)
- [Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA](https://mlsanity.com/paper/2408.09235/) (2024)
- [Revisiting Self-Training for Few-Shot Learning of Language Model](https://mlsanity.com/paper/2110.01256/) (2021)
- [ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding](https://mlsanity.com/paper/2411.05049/) (2024)
- [CC-Riddle: A Question Answering Dataset of Chinese Character Riddles](https://mlsanity.com/paper/2206.13778/) (2022)
- [RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation](https://mlsanity.com/paper/2409.16383/) (2024)
- [The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddles](https://mlsanity.com/paper/2511.00960/) (2025)
- [Adaptive Originality Filtering: Rejection Based Prompting and RiddleScore for Culturally Grounded Multilingual Riddle Generation](https://mlsanity.com/paper/2508.18709/) (2025)
- [BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali](https://mlsanity.com/paper/2511.20399/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) show impressive performance on many NLP benchmarks, yet their ability to reason in figurative, culturally grounded, and low-resource settings remains underexplored. We address this gap for Bangla by introducing BanglaRiddleEval, a benchmark of 1,244 traditional Bangla riddles instantiated across four tasks (4,976 riddle-task artifacts in total). Using an LLM-based pipeline, we generate Chain-of-Thought explanations, semantically coherent distractors, and fine-grained ambiguity annotations, and evaluate a diverse suite of open-source and closed-source models under different prompting strategies. Models achieve moderate semantic overlap on generative QA but low correctness, MCQ accuracy peaks at only about 56% versus an 83% human baseline, and ambiguity resolution ranges from roughly 26% to 68%, with high-quality explanations confined to the strongest models. These results show that current LLMs capture some cues needed for Bangla riddle reasoning but remain far from human-level performance, establishing BanglaRiddleEval as a challenging new benchmark for low-resource figurative reasoning. All data, code, and evaluation scripts are available on GitHub: https://github.com/Labib1610/BanglaRiddleEval.