# UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions

arXiv: https://arxiv.org/abs/2404.13343 · HTML: https://mlsanity.com/paper/2404.13343/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 16 references that are in the map.
- First posted on arXiv 20 Apr 2024.

## Builds on (most cited first, 3 of 3)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Diversity-Promoting Ensemble for Medical Image Segmentation](https://mlsanity.com/paper/2210.12388/) (2022)

## Built on by (most cited first, 3 of 3)

- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review](https://mlsanity.com/paper/2509.23486/) (2025)
- [Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling](https://mlsanity.com/paper/2608.04554/) (2026)

## Abstract (arXiv)

This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Our approach is based on augmenting the dataset with answers from zero-shot LLMs (Falcon, Meditron, Mistral) and employing transformer-based models based on six alternative feature combinations. The results suggest that predicting the difficulty of questions is more challenging. Notably, our top performing methods consistently include the question text, and benefit from the variability of LLM answers, highlighting the potential of LLMs for improving automated assessment in medical licensing exams. We make our code available https://github.com/ana-rogoz/BEA-2024.