# Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint

arXiv: https://arxiv.org/abs/2505.23759 · HTML: https://mlsanity.com/paper/2505.23759/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://mlsanity.com/paper/2311.17049/) (2023)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles](https://mlsanity.com/paper/2308.10855/) (2023)
- [TULIP: Towards Unified Language-Image Pretraining](https://mlsanity.com/paper/2503.15485/) (2025)
- [Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models](https://mlsanity.com/paper/2406.12546/) (2024)
- [REBUS: A Robust Evaluation Benchmark of Understanding Symbols](https://mlsanity.com/paper/2401.05604/) (2024)
- [Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text](https://mlsanity.com/paper/2112.00800/) (2021)
- [COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes](https://mlsanity.com/paper/2409.04053/) (2024)

## Built on by (most cited first, 3 of 3)

- [PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts](https://mlsanity.com/paper/2506.06211/) (2025)
- [Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Models](https://mlsanity.com/paper/2510.02780/) (2025)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs). Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns. In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels"). We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.