# Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses

arXiv: https://arxiv.org/abs/2408.00584 · HTML: https://mlsanity.com/paper/2408.00584/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Fine-tuning Large Language Models with Sequential Instructions](https://mlsanity.com/paper/2403.07794/) (2024)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [Automated Crossword Solving](https://mlsanity.com/paper/2205.09665/) (2022)
- [Missed Connections: Lateral Thinking Puzzles for Large Language Models](https://mlsanity.com/paper/2404.11730/) (2024)
- [Are LLMs Good Cryptic Crossword Solvers?](https://mlsanity.com/paper/2403.12094/) (2024)
- [Italian Crossword Generator: Enhancing Education through Interactive Word Puzzles](https://mlsanity.com/paper/2311.15723/) (2023)
- [Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles](https://mlsanity.com/paper/2404.06186/) (2024)
- [Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark](https://mlsanity.com/paper/2406.17535/) (2024)
- [Finding the optimal human strategy for Wordle using maximum correct letter probabilities and reinforcement learning](https://mlsanity.com/paper/2202.00557/) (2022)
- [A Turkish Educational Crossword Puzzle Generator](https://mlsanity.com/paper/2405.07035/) (2024)
- [The WebCrow French Crossword Solver](https://mlsanity.com/paper/2311.15626/) (2023)

## Built on by (most cited first, 2 of 2)

- [left|,circlearrowright,boxedBUS,right|: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles](https://mlsanity.com/paper/2511.01340/) (2025)
- ["Don't Say It!": Constraints, Compliance, and Communication when Language Models Play Taboo](https://mlsanity.com/paper/2607.00601/) (2026)

## Abstract (arXiv)

Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.