# Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Models

arXiv: https://arxiv.org/abs/2510.02780 · HTML: https://mlsanity.com/paper/2510.02780/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 11 of 11)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models](https://mlsanity.com/paper/2309.04461/) (2023)
- [LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles](https://mlsanity.com/paper/2308.10855/) (2023)
- [BRAINTEASER: Lateral Thinking Puzzles for Large Language Models](https://mlsanity.com/paper/2310.05057/) (2023)
- [GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs](https://mlsanity.com/paper/2311.04901/) (2023)
- [REBUS: A Robust Evaluation Benchmark of Understanding Symbols](https://mlsanity.com/paper/2401.05604/) (2024)
- [PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts](https://mlsanity.com/paper/2506.06211/) (2025)
- [Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint](https://mlsanity.com/paper/2505.23759/) (2025)

## Built on by (most cited first, 1 of 1)

- [Eye-Q: A Multilingual Benchmark for Visual Word Puzzle Solving and Image-to-Phrase Reasoning](https://mlsanity.com/paper/2601.03400/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) excel at many multimodal tasks, yet their cognitive processes remain opaque on complex lateral thinking challenges like rebus puzzles. While recent work has demonstrated these models struggle significantly with rebus puzzle solving, the underlying reasoning processes and failure patterns remain largely unexplored. We address this gap through a comprehensive explainability analysis that moves beyond performance metrics to understand how VLMs approach these complex lateral thinking challenges. Our study contributes a systematically annotated dataset of 221 rebus puzzles across six cognitive categories, paired with an evaluation framework that separates reasoning quality from answer correctness. We investigate three prompting strategies designed to elicit different types of explanatory processes and reveal critical insights into VLM cognitive processes. Our findings demonstrate that reasoning quality varies dramatically across puzzle categories, with models showing systematic strengths in visual composition while exhibiting fundamental limitations in absence interpretation and cultural symbolism. We also discover that prompting strategy substantially influences both cognitive approach and problem-solving effectiveness, establishing explainability as an integral component of model performance rather than a post-hoc consideration.