# Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty

arXiv: https://arxiv.org/abs/2502.17785 · HTML: https://mlsanity.com/paper/2502.17785/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 9 of 9)

- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education](https://mlsanity.com/paper/2410.21287/) (2024)
- [SPL: A Socratic Playground for Learning Powered by Large Language Model](https://mlsanity.com/paper/2406.13919/) (2024)
- [Predicting Learning Performance with Large Language Models: A Study in Adult Literacy](https://mlsanity.com/paper/2403.14668/) (2024)
- [Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models](https://mlsanity.com/paper/2404.07720/) (2024)

## Built on by (most cited first, 3 of 3)

- [Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?](https://mlsanity.com/paper/2510.25064/) (2025)
- [Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring](https://mlsanity.com/paper/2605.12398/) (2026)
- [An LLM-Based System for Argument Mining](https://mlsanity.com/paper/2605.13793/) (2026)

## Abstract (arXiv)

Reading comprehension is a key for individual success, yet the assessment of question difficulty remains challenging due to the extensive human annotation and large-scale testing required by traditional methods such as linguistic analysis and Item Response Theory (IRT). While these robust approaches provide valuable insights, their scalability is limited. There is potential for Large Language Models (LLMs) to automate question difficulty estimation; however, this area remains underexplored. Our study investigates the effectiveness of LLMs, specifically OpenAI's GPT-4o and o1, in estimating the difficulty of reading comprehension questions using the Study Aid and Reading Assessment (SARA) dataset. We evaluated both the accuracy of the models in answering comprehension questions and their ability to classify difficulty levels as defined by IRT. The results indicate that, while the models yield difficulty estimates that align meaningfully with derived IRT parameters, there are notable differences in their sensitivity to extreme item characteristics. These findings suggest that LLMs can serve as the scalable method for automated difficulty assessment, particularly in dynamic interactions between learners and Adaptive Instructional Systems (AIS), bridging the gap between traditional psychometric techniques and modern AIS for reading comprehension and paving the way for more adaptive and personalized educational assessments.