# Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

arXiv: https://arxiv.org/abs/2103.01242 · HTML: https://mlsanity.com/paper/2103.01242/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [iSarcasm: A Dataset of Intended Sarcasm](https://mlsanity.com/paper/1911.03123/) (2019)

## Built on by (most cited first, 15 of 15)

- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [ChatHaruhi: Reviving Anime Character in Reality via Large Language Model](https://mlsanity.com/paper/2308.09597/) (2023)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning](https://mlsanity.com/paper/2410.04691/) (2024)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation](https://mlsanity.com/paper/2504.00043/) (2025)
- [Automated Crossword Solving](https://mlsanity.com/paper/2205.09665/) (2022)
- [Are LLMs Good Cryptic Crossword Solvers?](https://mlsanity.com/paper/2403.12094/) (2024)
- [CC-Riddle: A Question Answering Dataset of Chinese Character Riddles](https://mlsanity.com/paper/2206.13778/) (2022)
- [Language Models are Crossword Solvers](https://mlsanity.com/paper/2406.09043/) (2024)
- [A Major Obstacle for NLP Research: Let's Talk about Time Allocation!](https://mlsanity.com/paper/2211.16858/) (2022)
- [Proving that Cryptic Crossword Clue Answers are Correct](https://mlsanity.com/paper/2407.08824/) (2024)
- [What Makes Cryptic Crosswords Challenging for LLMs?](https://mlsanity.com/paper/2412.09012/) (2024)

## Abstract (arXiv)

Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).