# Estimating Item Difficulty with Large Language Models as Experts

arXiv: https://arxiv.org/abs/2605.18562 · HTML: https://mlsanity.com/paper/2605.18562/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 72 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation](https://mlsanity.com/paper/2504.14716/) (2025)
- [Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons](https://mlsanity.com/paper/2405.05894/) (2024)
- [R2DE: a NLP approach to estimating IRT parameters of newly generated questions](https://mlsanity.com/paper/2001.07569/) (2020)
- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [Introducing a framework to assess newly created questions with Natural Language Processing](https://mlsanity.com/paper/2004.13530/) (2020)
- [Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction](https://mlsanity.com/paper/2512.18880/) (2025)
- [Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms](https://mlsanity.com/paper/2504.08804/) (2025)
- [Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI](https://mlsanity.com/paper/2511.06078/) (2025)
- [Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations](https://mlsanity.com/paper/2601.09953/) (2026)
- [Estimating problem difficulty without ground truth using Large Language Model comparisons](https://mlsanity.com/paper/2512.14220/) (2025)
- [Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?](https://mlsanity.com/paper/2507.08232/) (2025)
- [Finetuning LLMs for Comparative Assessment Tasks](https://mlsanity.com/paper/2409.15979/) (2024)
- [NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty](https://mlsanity.com/paper/2508.03294/) (2025)

## Built on by (most cited first, 1 of 1)

- [Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques](https://mlsanity.com/paper/2608.06609/) (2026)

## Abstract (arXiv)

Accurate estimates of item difficulty are essential for valid assessment and effective adaptive learning. However, for newly created tasks, response data are typically unavailable. Pretesting and expert judgement can be costly and slow, while machine learning methods often require large labelled training datasets. Recent work suggests that large language models (LLMs) may help. However, there is limited evidence on the elicitation procedures and prompt configurations used to emulate experts for difficulty estimation. This study addresses this gap by evaluating three off-the-shelf LLMs as difficulty raters for newly created items without access to response data. Using an item bank from an online learning system, the study examined 6 domains of primary-school mathematics, with empirical difficulty estimates treated as empirical reference. The study used a full factorial design crossing three factors: judgement format (absolute vs pairwise), decision type (hard decisions vs token-probability-based estimates), and prompting strategy (zero-shot vs few-shot). LLM-derived difficulty estimates were compared with empirical difficulties using Spearman rank correlations. Across domains, LLM-based estimates exhibited moderate to strong positive correlations with empirical item difficulties. For simpler arithmetic tasks, some configurations approached the upper end of the accuracy range reported for human experts in previous research. Pairwise comparison consistently outperformed absolute judgement in the absence of additional refinements. However, when token-level probabilities were incorporated and examples of items with known empirical difficulty were provided, the absolute judgement configuration likewise demonstrated moderate-to-high alignment. The study positions LLMs as a promising tool for initial item calibration and offers insights into effective workflow configuration.