mlsanityData to 7 Oct 2026

SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction

arXiv 2507.05129 ↗First posted 7 Jul 2025cs.CL cs.CY cs.LGcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Item (question) difficulties play a crucial role in educational assessments, enabling accurate and efficient assessment of student abilities and personalization to maximize learning outcomes. Traditionally, estimating item difficulties can be costly, requiring real students to respond to items, followed by fitting an item response theory (IRT) model to get difficulty estimates. This approach cannot be applied to the cold-start setting for previously unseen items either. In this work, we present SMART (Simulated Students Aligned with IRT), a novel method for aligning simulated students with instructed ability, which can then be used in simulations to predict the difficulty of open-ended items. We achieve this alignment using direct preference optimization (DPO), where we form preference pairs based on how likely responses are under a ground-truth IRT model. We perform a simulation by generating thousands of responses, evaluating them with a large language model (LLM)-based scoring model, and fit the resulting data to an IRT model to obtain item difficulty estimates. Through extensive experiments on two real-world student response datasets, we show that SMART outperforms other item difficulty prediction methods by leveraging its improved ability alignment.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 71 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

26 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LookAlike: Consistent Distractor Generation in Math MCQs3 May

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  7. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  8. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  9. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  10. 2024Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMscited by 44
  11. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  12. 2019The continuous Bernoulli: fixing a pervasive error in variational autoencoderscited by 40
13 more
  1. 2019Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowdscited by 40
  2. 2025Training LLM-based Tutors to Improve Student Learning Outcomes in Dialoguescited by 40
  3. 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
  4. 2025Mixture-of-Personas Language Models for Population Simulationcited by 20
  5. 2025Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agentscited by 19
  6. 2024Psychometric Alignment: Capturing Human Knowledge Distributions via Language Modelscited by 18
  7. 2023Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiencycited by 15
  8. 2025Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMscited by 14
  9. 2023A quantitative study of NLP approaches to question difficulty estimationcited by 12
  10. 2023Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rankcited by 12
  11. 2025Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repositorycited by 5
  12. 2024Student Data Paradox and Curious Case of Single Student-Tutor Model: Regressive Side Effects of Training LLMs for Personalized Learningcited by 4
  13. 2025Cyborg Data: Merging Human with AI Generated Training Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 19.

21 papers.

Sort:

By area:

  1. 2026Simulated Students in Tutoring Dialogues: Substance or Illusion?cited by 14
  2. 2026Towards Valid Student Simulation with Large Language Modelscited by 7
  3. 2026KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Taskscited by 7
  4. 2026BRIDGE: Predicting Human Task Completion Time From Model Performancecited by 6
  5. 2025RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMscited by 5
  6. 2026Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulationscited by 5
  7. 2026FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMscited by 4
  8. 2026PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedbackcited by 2
  9. 2025RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoningcited by 1
  10. 2026Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Modelscited by 1
  11. 2026SWIM: Student Writing Simulation via Proficiency-Conditioned Generationcited by 1
  12. 2026CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problemscited by 0
9 more
  1. 2026Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theorycited by 0
  2. 2026Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialoguescited by 0
  3. 2026MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profilingcited by 0
  4. 2026VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrievalcited by 0
  5. 2026Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approachcited by 0
  6. 2026Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiquescited by 0
  7. 2026Multimodal Item Parameter Estimation using Simulated Response Probabilitiecited by 0
  8. 2026A Dataset for Modeling Iterative Problem-Solvingcited by 0
  9. 2026From Learner Behavior to Reusable Skills for Effective and Efficient Learner Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
  2. 2026Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling
  3. 2025Reliable and Efficient Amortized Model-based Evaluation
  4. 2026KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
  5. 2025Scaling Item-to-Standard Alignment with Large Language Models: Accuracy, Limits, and Solutions
  6. 2024IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
  7. 2025Socratic Students: Teaching Language Models to Learn by Asking Questions
  8. 2025LLM-driven Effective Knowledge Tracing by Integrating Dual-channel Difficulty