mlsanityData to 7 Oct 2026

Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language

arXiv 2103.01242 ↗First posted 1 Mar 2021cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease. We present Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced. Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a misleading surface reading, whose solving requires disambiguating semantic, syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues pose a challenge even for experienced solvers, though top-tier experts can solve them with almost 100% accuracy. Cryptonite is a challenging task for current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6% accuracy, on par with the accuracy of a rule-based clue solver (8.6%).

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 7 of its 29 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

7 of 29 references.

References · 7

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  6. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  7. 2019iSarcasm: A Dataset of Intended Sarcasmcited by 39

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 2; 2024: 6; 2025: 1.

15 papers.

Sort:

By area:

  1. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  2. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  4. 2022What do tokens know about their characters and how do they know it?cited by 36
  5. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  6. 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
  7. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  8. 2022Automated Crossword Solvingcited by 9
  9. 2025CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generationcited by 9
  10. 2024Are LLMs Good Cryptic Crossword Solvers?cited by 7
  11. 2022CC-Riddle: A Question Answering Dataset of Chinese Character Riddlescited by 6
  12. 2024Language Models are Crossword Solverscited by 5
3 more
  1. 2022A Major Obstacle for NLP Research: Let's Talk about Time Allocation!cited by 1
  2. 2024Proving that Cryptic Crossword Clue Answers are Correctcited by 0
  3. 2024What Makes Cryptic Crosswords Challenging for LLMs?cited by 0