mlsanityData to 7 Oct 2026

Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach

arXiv 2607.26317 ↗First posted 28 Jul 2026cs.CY cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Psychometric calibration for educational tests typically requires costly human response data. Large language models (LLMs) simulated examinees offer a promising route to early calibration, but their responses are too accurate and too uniform. We propose Cognitive Diagnostic Profiling (CDP), a zero-shot framework that prompts LLMs to simulate plausible examinees with diverse cognitive profiles: binary attribute-mastery patterns are rendered as natural-language profiles and sampled under an uninformative or an informative distribution. Using the Tatsuoka fraction-subtraction dataset (536 examinees, 15 items, five attributes), we evaluated eight LLM configurations under no-profile, uninformative-CDP, and informative-CDP conditions, assessing alignment with human examinees at the ability-distribution, mastery-profile, and item-difficulty levels. CDP improved all three levels: distributional overlap rose across configurations; weighted correlations between profile-level scores and human profile expectations reached 0.92 to 0.98; and item-difficulty recovery improved in rank order and absolute alignment, most for reasoning-enabled models; in the strongest case, Gemini 3.0 Flash (Thinking), one-parameter logistic (1PL) difficulty Spearman correlations rose from 0.24 to 0.86 and 0.90 and the root-mean-square error (RMSE) fell from 6.31 to 1.30 and 0.90; the informative condition helped most where profile-level alignment was strong. CDP brings LLM-simulated examinees into closer psychometric alignment with human examinees, making them practical for operational test development.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 52 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

20 of 52 references.

References · 20

  1. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  2. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 56
  3. 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
  4. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
  5. 2025SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Predictioncited by 21
  6. 2025Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agentscited by 19
  7. 2024Psychometric Alignment: Capturing Human Knowledge Distributions via Language Modelscited by 18
  8. 2025Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?cited by 14
  9. 2025Knowledge is Power: Harnessing Large Language Models for Enhanced Cognitive Diagnosiscited by 10
  10. 2025Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Reviewcited by 10
  11. 2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Predictioncited by 7
  12. 2025Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AIcited by 5
8 more
  1. 2026Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulationscited by 5
  2. 2024Student Data Paradox and Curious Case of Single Student-Tutor Model: Regressive Side Effects of Training LLMs for Personalized Learningcited by 4
  3. 2025QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulationcited by 3
  4. 2025Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Contextcited by 3
  5. 2025Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?cited by 3
  6. 2026Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Modelscited by 1
  7. 2026Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimationcited by 1
  8. 2026Estimating Exam Item Difficulty with LLMs: A Benchmark on Brazil's ENEM Corpuscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis
  2. 2025In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs
  3. 2025From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests
  4. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
  5. 2026Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models
  6. 2025Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy
  7. 2026Large Language Models as Simulative Agents for Neurodivergent Adult Psychometric Profiles
  8. 2025Scaling Item-to-Standard Alignment with Large Language Models: Accuracy, Limits, and Solutions