# Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review

arXiv: https://arxiv.org/abs/2509.23486 · HTML: https://mlsanity.com/paper/2509.23486/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 71 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [A quantitative study of NLP approaches to question difficulty estimation](https://mlsanity.com/paper/2305.10236/) (2023)
- [Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation](https://mlsanity.com/paper/2412.11831/) (2024)
- [Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms](https://mlsanity.com/paper/2504.08804/) (2025)
- [Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository](https://mlsanity.com/paper/2502.20663/) (2025)
- [UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions](https://mlsanity.com/paper/2404.13343/) (2024)

## Built on by (most cited first, 10 of 10)

- [Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction](https://mlsanity.com/paper/2512.18880/) (2025)
- [Towards Valid Student Simulation with Large Language Models](https://mlsanity.com/paper/2601.05473/) (2026)
- [LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment](https://mlsanity.com/paper/2606.18709/) (2026)
- [Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction](https://mlsanity.com/paper/2606.28186/) (2026)
- [Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.02580/) (2026)
- [RoMathExam: A Longitudinal Dataset of Romanian Math Exams (1895-2025) with a Seven-Decade Core (1957-2025)](https://mlsanity.com/paper/2604.16392/) (2026)
- [Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning](https://mlsanity.com/paper/2605.16991/) (2026)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling](https://mlsanity.com/paper/2608.04554/) (2026)
- [Multimodal Item Parameter Estimation using Simulated Response Probabilitie](https://mlsanity.com/paper/2608.10154/) (2026)

## Abstract (arXiv)

Item difficulty plays a crucial role in test performance, interpretability of scores, and equity for all test-takers, especially in large-scale assessments. Traditional approaches to item difficulty modeling rely on field testing and classical test theory (CTT)-based item analysis or item response theory (IRT) calibration, which can be time-consuming and costly. To overcome these challenges, text-based approaches leveraging machine learning and language models, have emerged as promising alternatives. This paper reviews and synthesizes 37 articles on automated item difficulty prediction in large-scale assessment settings published through May 2025. For each study, we delineate the dataset, difficulty parameter, subject domain, item type, number of items, training and test data split, input, features, model, evaluation criteria, and model performance outcomes. Results showed that although classic machine learning models remain relevant due to their interpretability, state-of-the-art language models, using both small and large transformer-based architectures, can capture syntactic and semantic patterns without the need for manual feature engineering. Uniquely, model performance outcomes were summarized to serve as a benchmark for future research and overall, text-based methods have the potential to predict item difficulty with root mean square error (RMSE) as low as 0.165, Pearson correlation as high as 0.87, and accuracy as high as 0.806. The review concludes by discussing implications for practice and outlining future research directions for automated item difficulty modeling.