Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses
Not on the map. Why?
Abstract (arXiv)
Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 1 Aug 2024.
Builds on
17 of 42 references.
References · 17
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
- 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 45
- 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
- 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
- 2022Automated Crossword Solvingcited by 9
- 2024Missed Connections: Lateral Thinking Puzzles for Large Language Modelscited by 9
- 2024Are LLMs Good Cryptic Crossword Solvers?cited by 7
- 2023Italian Crossword Generator: Enhancing Education through Interactive Word Puzzlescited by 6
5 more
- 2024Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzlescited by 5
- 2024Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmarkcited by 5
- 2022Finding the optimal human strategy for Wordle using maximum correct letter probabilities and reinforcement learningcited by 4
- 2024A Turkish Educational Crossword Puzzle Generatorcited by 4
- 2023The WebCrow French Crossword Solvercited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.