# A quantitative study of NLP approaches to question difficulty estimation

arXiv: https://arxiv.org/abs/2305.10236 · HTML: https://mlsanity.com/paper/2305.10236/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Enhancing Item Response Theory for Cognitive Diagnosis](https://mlsanity.com/paper/1905.10957/) (2019)
- [R2DE: a NLP approach to estimating IRT parameters of newly generated questions](https://mlsanity.com/paper/2001.07569/) (2020)
- [Introducing a framework to assess newly created questions with Natural Language Processing](https://mlsanity.com/paper/2004.13530/) (2020)
- [Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model](https://mlsanity.com/paper/2110.06560/) (2021)
- [Manipulating the Difficulty of C-Tests](https://mlsanity.com/paper/1906.06905/) (2019)
- [Exercise Hierarchical Feature Enhanced Knowledge Tracing](https://mlsanity.com/paper/2011.09867/) (2020)
- [Development and Evaluation of a Personalized Computer-aided Question Generation for English Learners to Improve Proficiency and Correct Mistakes](https://mlsanity.com/paper/1808.09732/) (2018)

## Built on by (most cited first, 13 of 13)

- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review](https://mlsanity.com/paper/2509.23486/) (2025)
- [Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction](https://mlsanity.com/paper/2512.18880/) (2025)
- [Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations](https://mlsanity.com/paper/2601.09953/) (2026)
- [LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment](https://mlsanity.com/paper/2606.18709/) (2026)
- [An Information-Theoretic Approach to Analyze NLP Classification Tasks](https://mlsanity.com/paper/2402.00978/) (2024)
- [Active Learning to Guide Labeling Efforts for Question Difficulty Estimation](https://mlsanity.com/paper/2409.09258/) (2024)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)
- [A Conceptual Framework for AI Capability Evaluations](https://mlsanity.com/paper/2506.18213/) (2025)
- [Ordinality in Discrete-level Question Difficulty Estimation: Introducing Balanced DRPS and OrderedLogitNN](https://mlsanity.com/paper/2507.00736/) (2025)
- [One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries](https://mlsanity.com/paper/2603.11545/) (2026)
- [A Unified Response Framework for Questions and Tutoring Dialogues](https://mlsanity.com/paper/2610.19532/) (2026)

## Abstract (arXiv)

Recent years witnessed an increase in the amount of research on the task of Question Difficulty Estimation from Text QDET with Natural Language Processing (NLP) techniques, with the goal of targeting the limitations of traditional approaches to question calibration. However, almost the entirety of previous research focused on single silos, without performing quantitative comparisons between different models or across datasets from different educational domains. In this work, we aim at filling this gap, by quantitatively analyzing several approaches proposed in previous research, and comparing their performance on three publicly available real world datasets containing questions of different types from different educational domains. Specifically, we consider reading comprehension Multiple Choice Questions (MCQs), science MCQs, and math questions. We find that Transformer based models are the best performing across different educational domains, with DistilBERT performing almost as well as BERT, and that they outperform other approaches even on smaller datasets. As for the other models, the hybrid ones often outperform the ones based on a single type of features, the ones based on linguistic features perform well on reading comprehension questions, while frequency based features (TF-IDF) and word embeddings (word2vec) perform better in domain knowledge assessment.