# Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction

arXiv: https://arxiv.org/abs/2512.18880 · HTML: https://mlsanity.com/paper/2512.18880/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 21 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals](https://mlsanity.com/paper/2406.10881/) (2024)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations](https://mlsanity.com/paper/2402.15062/) (2024)
- [LLMs and Memorization: On Quality and Specificity of Copyright Compliance](https://mlsanity.com/paper/2405.18492/) (2024)
- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?](https://mlsanity.com/paper/2506.09796/) (2025)
- [A quantitative study of NLP approaches to question difficulty estimation](https://mlsanity.com/paper/2305.10236/) (2023)
- [Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review](https://mlsanity.com/paper/2509.23486/) (2025)
- [Modeling Student Learning with 3.8 Million Program Traces](https://mlsanity.com/paper/2510.05056/) (2025)
- [Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld's Episode Theory](https://mlsanity.com/paper/2509.14662/) (2025)
- [Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation](https://mlsanity.com/paper/2412.11831/) (2024)
- [Constrained Probabilistic Mask Learning for Task-specific Undersampled MRI Reconstruction](https://mlsanity.com/paper/2305.16376/) (2023)

## Built on by (most cited first, 7 of 7)

- [Towards Valid Student Simulation with Large Language Models](https://mlsanity.com/paper/2601.05473/) (2026)
- [LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment](https://mlsanity.com/paper/2606.18709/) (2026)
- [Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction](https://mlsanity.com/paper/2606.28186/) (2026)
- [BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulation](https://mlsanity.com/paper/2602.13280/) (2026)
- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling](https://mlsanity.com/paper/2608.04554/) (2026)

## Abstract (arXiv)

Accurate estimation of item (question or task) difficulty is critical for educational assessment but suffers from the cold start problem. While Large Language Models demonstrate superhuman problem-solving capabilities, it remains an open question whether they can perceive the cognitive struggles of human learners. In this work, we present a large-scale empirical analysis of Human-AI Difficulty Alignment for over 20 models across diverse domains such as medical knowledge and mathematical reasoning. Our findings reveal a systematic misalignment where scaling up model size is not reliably helpful; instead of aligning with humans, models converge toward a shared machine consensus. We observe that high performance often impedes accurate difficulty estimation, as models struggle to simulate the capability limitations of students even when being explicitly prompted to adopt specific proficiency levels. Furthermore, we identify a critical lack of introspection, as models fail to predict their own limitations. These results suggest that general problem-solving capability does not imply an understanding of human cognitive struggles, highlighting the challenge of using current models for automated difficulty prediction.