# Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository

arXiv: https://arxiv.org/abs/2502.20663 · HTML: https://mlsanity.com/paper/2502.20663/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 36 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration](https://mlsanity.com/paper/2410.21033/) (2024)

## Built on by (most cited first, 5 of 5)

- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review](https://mlsanity.com/paper/2509.23486/) (2025)
- [Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?](https://mlsanity.com/paper/2510.25064/) (2025)
- [Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation](https://mlsanity.com/paper/2602.00034/) (2026)
- [It's How You Ask: Gender-Associated Linguistic Bias in LLMs](https://mlsanity.com/paper/2608.13328/) (2026)

## Abstract (arXiv)

Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2018-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.59 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.