# Introducing a framework to assess newly created questions with Natural Language Processing

arXiv: https://arxiv.org/abs/2004.13530 · HTML: https://mlsanity.com/paper/2004.13530/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 8 of 8)

- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [A quantitative study of NLP approaches to question difficulty estimation](https://mlsanity.com/paper/2305.10236/) (2023)
- [Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms](https://mlsanity.com/paper/2504.08804/) (2025)
- [Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training](https://mlsanity.com/paper/2301.07558/) (2023)
- [Proactive and Reactive Engagement of Artificial Intelligence Methods for Education: A Review](https://mlsanity.com/paper/2301.10231/) (2023)
- [Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation](https://mlsanity.com/paper/2602.00034/) (2026)
- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [Transferable Student Performance Modeling for Intelligent Tutoring Systems](https://mlsanity.com/paper/2202.03980/) (2022)

## Abstract (arXiv)

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.