# Are LLMs Good Cryptic Crossword Solvers?

arXiv: https://arxiv.org/abs/2403.12094 · HTML: https://mlsanity.com/paper/2403.12094/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Pun Generation with Surprise](https://mlsanity.com/paper/1904.06828/) (2019)
- [Pun-GAN: Generative Adversarial Network for Pun Generation](https://mlsanity.com/paper/1910.10950/) (2019)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language](https://mlsanity.com/paper/2103.01242/) (2021)

## Built on by (most cited first, 7 of 7)

- [CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation](https://mlsanity.com/paper/2504.00043/) (2025)
- [PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles](https://mlsanity.com/paper/2510.06475/) (2025)
- [Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses](https://mlsanity.com/paper/2408.00584/) (2024)
- [Do You Get the Hint? Benchmarking LLMs on the Board Game Concept](https://mlsanity.com/paper/2510.13271/) (2025)
- [left|,circlearrowright,boxedBUS,right|: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles](https://mlsanity.com/paper/2511.01340/) (2025)
- [What Makes Cryptic Crosswords Challenging for LLMs?](https://mlsanity.com/paper/2412.09012/) (2024)
- [A Reasoning-Based Approach to Cryptic Crossword Clue Solving](https://mlsanity.com/paper/2506.04824/) (2025)

## Abstract (arXiv)

Cryptic crosswords are puzzles that rely not only on general knowledge but also on the solver's ability to manipulate language on different levels and deal with various types of wordplay. Previous research suggests that solving such puzzles is a challenge even for modern NLP models. However, the abilities of large language models (LLMs) have not yet been tested on this task. In this paper, we establish the benchmark results for three popular LLMs -- LLaMA2, Mistral, and ChatGPT -- showing that their performance on this task is still far from that of humans.