# Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment

arXiv: https://arxiv.org/abs/2403.01456 · HTML: https://mlsanity.com/paper/2403.01456/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Large-scale Cloze Test Dataset Created by Teachers](https://mlsanity.com/paper/1711.03225/) (2017)
- [Automatic Distractor Generation for Multiple Choice Questions in Standard Tests](https://mlsanity.com/paper/2011.13100/) (2020)
- [Generating Distractors for Reading Comprehension Questions from Real Examinations](https://mlsanity.com/paper/1809.02768/) (2018)
- [Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questions](https://mlsanity.com/paper/2004.09853/) (2020)
- [A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies](https://mlsanity.com/paper/2010.05384/) (2020)
- [py-irt: A Scalable Item Response Theory Library for Python](https://mlsanity.com/paper/2203.01282/) (2022)
- [Empowering Active Learning to Jointly Optimize System and User Demands](https://mlsanity.com/paper/2005.04470/) (2020)
- [Manipulating the Difficulty of C-Tests](https://mlsanity.com/paper/1906.06905/) (2019)
- [Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers](https://mlsanity.com/paper/2204.07237/) (2022)

## Built on by (most cited first, 1 of 1)

- [Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs](https://mlsanity.com/paper/2507.01334/) (2025)

## Abstract (arXiv)

Item difficulty plays a crucial role in adaptive testing. However, few works have focused on generating questions of varying difficulty levels, especially for multiple-choice (MC) cloze tests. We propose training pre-trained language models (PLMs) as surrogate models to enable item response theory (IRT) assessment, avoiding the need for human test subjects. We also propose two strategies to control the difficulty levels of both the gaps and the distractors using ranking rules to reduce invalid distractors. Experimentation on a benchmark dataset demonstrates that our proposed framework and methods can effectively control and evaluate the difficulty levels of MC cloze tests.