# Large-scale Cloze Test Dataset Created by Teachers

arXiv: https://arxiv.org/abs/1711.03225 · HTML: https://mlsanity.com/paper/1711.03225/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2017.

## Built on by (most cited first, 27 of 27)

- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench](https://mlsanity.com/paper/2410.22108/) (2024)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.15910/) (2025)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [EduQG: A Multi-format Multiple Choice Dataset for the Educational Domain](https://mlsanity.com/paper/2210.06104/) (2022)
- [Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words](https://mlsanity.com/paper/2202.12142/) (2022)
- [A Survey on non-English Question Answering Dataset](https://mlsanity.com/paper/2112.13634/) (2021)
- [Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers](https://mlsanity.com/paper/2204.07237/) (2022)
- [ViMMRC 2.0 -- Enhancing Machine Reading Comprehension on Vietnamese Literature Text](https://mlsanity.com/paper/2303.18162/) (2023)
- [CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model](https://mlsanity.com/paper/2403.10326/) (2024)
- [Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension](https://mlsanity.com/paper/2112.06494/) (2021)
- [LRG at SemEval-2021 Task 4: Improving Reading Comprehension with Abstract Words using Augmentation, Linguistic Features and Voting](https://mlsanity.com/paper/2102.12255/) (2021)
- [Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading](https://mlsanity.com/paper/2310.13409/) (2023)
- [Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment](https://mlsanity.com/paper/2403.01456/) (2024)
- [FairytaleQA Translated: Enabling Educational Question and Answer Generation in Less-Resourced Languages](https://mlsanity.com/paper/2406.04233/) (2024)
- [InDEX: Indonesian Idiom and Expression Dataset for Cloze Test](https://mlsanity.com/paper/2211.13376/) (2022)
- [Cross-corpus Readability Compatibility Assessment for English Texts](https://mlsanity.com/paper/2306.09704/) (2023)
- [Recent Advances in Multi-Choice Machine Reading Comprehension: A Survey on Methods and Datasets](https://mlsanity.com/paper/2408.02114/) (2024)
- [Difficulty-Controllable Cloze Question Distractor Generation](https://mlsanity.com/paper/2511.01526/) (2025)
- [ESACT: An End-to-End Sparse Accelerator for Compute-Intensive Transformers via Local Similarity](https://mlsanity.com/paper/2512.02403/) (2025)
- [Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs](https://mlsanity.com/paper/2602.15846/) (2026)
- [Beetle: A Bilingual Model Suite for Modelling Second-Language Processing](https://mlsanity.com/paper/2609.22633/) (2026)
- [The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale](https://mlsanity.com/paper/2609.26087/) (2026)

## Abstract (arXiv)

Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.