# Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs

arXiv: https://arxiv.org/abs/2503.08551 · HTML: https://mlsanity.com/paper/2503.08551/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 7 of 7)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [What Makes Math Word Problems Challenging for LLMs?](https://mlsanity.com/paper/2403.11369/) (2024)
- [R2DE: a NLP approach to estimating IRT parameters of newly generated questions](https://mlsanity.com/paper/2001.07569/) (2020)
- [Introducing a framework to assess newly created questions with Natural Language Processing](https://mlsanity.com/paper/2004.13530/) (2020)
- [Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT](https://mlsanity.com/paper/2412.10267/) (2024)
- [Balancing Test Accuracy and Security in Computerized Adaptive Testing](https://mlsanity.com/paper/2305.18312/) (2023)
- [UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions](https://mlsanity.com/paper/2404.13343/) (2024)

## Built on by (most cited first, 14 of 14)

- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review](https://mlsanity.com/paper/2509.23486/) (2025)
- [Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction](https://mlsanity.com/paper/2512.18880/) (2025)
- [KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks](https://mlsanity.com/paper/2601.06633/) (2026)
- [Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations](https://mlsanity.com/paper/2601.09953/) (2026)
- [FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs](https://mlsanity.com/paper/2602.00070/) (2026)
- [LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment](https://mlsanity.com/paper/2606.18709/) (2026)
- [From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings](https://mlsanity.com/paper/2607.07141/) (2026)
- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)
- [Optimizing Coverage and Difficulty in Reinforcement Learning for Quiz Composition](https://mlsanity.com/paper/2603.27695/) (2026)
- [MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling](https://mlsanity.com/paper/2605.16290/) (2026)
- [Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling](https://mlsanity.com/paper/2608.04554/) (2026)
- [Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques](https://mlsanity.com/paper/2608.06609/) (2026)

## Abstract (arXiv)

The difficulty of multiple-choice questions (MCQs) is a crucial factor for educational assessments. Predicting MCQ difficulty is challenging since it requires understanding both the complexity of reaching the correct option and the plausibility of distractors, i.e., incorrect options. In this paper, we propose a novel, two-stage method to predict the difficulty of MCQs. First, to better estimate the complexity of each MCQ, we use large language models (LLMs) to augment the reasoning steps required to reach each option. We use not just the MCQ itself but also these reasoning steps as input to predict the difficulty. Second, to capture the plausibility of distractors, we sample knowledge levels from a distribution to account for variation among students responding to the MCQ. This setup, inspired by item response theory (IRT), enable us to estimate the likelihood of students selecting each (both correct and incorrect) option. We align these predictions with their ground truth values, using a Kullback-Leibler (KL) divergence-based regularization objective, and use estimated likelihoods to predict MCQ difficulty. We evaluate our method on two real-world math MCQ and response datasets with ground truth difficulty values estimated using IRT. Experimental results show that our method outperforms all baselines, up to a 28.3% reduction in mean squared error and a 34.6% improvement in the coefficient of determination. We also qualitatively discuss how our novel method results in higher accuracy in predicting MCQ difficulty.