# left|,circlearrowright,boxedBUS,right|: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles

arXiv: https://arxiv.org/abs/2511.01340 · HTML: https://mlsanity.com/paper/2511.01340/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [REBUS: A Robust Evaluation Benchmark of Understanding Symbols](https://mlsanity.com/paper/2401.05604/) (2024)
- [Are LLMs Good Cryptic Crossword Solvers?](https://mlsanity.com/paper/2403.12094/) (2024)
- [Italian Crossword Generator: Enhancing Education through Interactive Word Puzzles](https://mlsanity.com/paper/2311.15723/) (2023)
- [Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles](https://mlsanity.com/paper/2404.06186/) (2024)
- [A Turkish Educational Crossword Puzzle Generator](https://mlsanity.com/paper/2405.07035/) (2024)
- [Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses](https://mlsanity.com/paper/2408.00584/) (2024)

## Built on by (most cited first, 1 of 1)

- [Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles](https://mlsanity.com/paper/2601.13705/) (2026)

## Abstract (arXiv)

Understanding Rebus Puzzles (Rebus Puzzles use pictures, symbols, and letters to represent words or phrases creatively) requires a variety of skills such as image recognition, cognitive skills, commonsense reasoning, multi-step reasoning, image-based wordplay, etc., making this a challenging task for even current Vision-Language Models. In this paper, we present left|,circlearrowright,boxedBUS,right|, a large and diverse benchmark of 1,333 English Rebus Puzzles containing different artistic styles and levels of difficulty, spread across 18 categories such as food, idioms, sports, finance, entertainment, etc. We also propose RebusDescProgICE, a model-agnostic framework which uses a combination of an unstructured description and code-based, structured reasoning, along with better, reasoning-based in-context example selection, improving the performance of Vision-Language Models on left|,circlearrowright,boxedBUS,right| by 2.1-4.1% and 20-30% using closed-source and open-source models respectively compared to Chain-of-Thought Reasoning.