mlsanityData to 14 Oct 2026

Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint

arXiv 2505.23759 ↗First posted 29 May 2025cs.CL cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs). Unlike traditional image captioning or question answering tasks, rebus solving requires multi-modal abstraction, symbolic reasoning, and a grasp of cultural, phonetic and linguistic puns. In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles, ranging from simple pictographic substitutions to spatially-dependent cues ("head" over "heels"). We analyze how different VLMs perform, and our findings reveal that while VLMs exhibit some surprising capabilities in decoding simple visual clues, they struggle significantly with tasks requiring abstract reasoning, lateral thinking, and understanding visual metaphors.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 17 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

11 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024REBUS: A Robust Evaluation Benchmark of Understanding Symbols11 Jan

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  3. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  4. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  5. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 61
  6. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
  7. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  8. 2024Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Modelscited by 9
  9. 2021Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Textcited by 6
  10. 2024COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehuntscited by 8
  2. 2025Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Modelscited by 1
  3. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025left|,circlearrowright,boxedBUS,right|: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles
  2. 2026Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning
  3. 2026Eye-Q: A Multilingual Benchmark for Visual Word Puzzle Solving and Image-to-Phrase Reasoning
  4. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles
  5. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning
  6. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
  7. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection
  8. 2025VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs