# Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis

arXiv: https://arxiv.org/abs/2407.10899 · HTML: https://mlsanity.com/paper/2407.10899/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [Automatic Generation of Socratic Subquestions for Teaching Math Word Problems](https://mlsanity.com/paper/2211.12835/) (2022)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Learning by Analogy: Diverse Questions Generation in Math Word Problem](https://mlsanity.com/paper/2306.09064/) (2023)
- [Leveraging generative artificial intelligence to simulate student learning behavior](https://mlsanity.com/paper/2310.19206/) (2023)
- [pyBKT: An Accessible Python Library of Bayesian Knowledge Tracing Models](https://mlsanity.com/paper/2105.00385/) (2021)
- [LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2402.14568/) (2024)
- [Math Multiple Choice Question Generation via Human-Large Language Model Collaboration](https://mlsanity.com/paper/2405.00864/) (2024)
- [Human-AI Collaboration Increases Skill Tagging Speed but Degrades Accuracy](https://mlsanity.com/paper/2403.02259/) (2024)
- [Automatic Generation of Word Problems for Academic Education via Natural Language Processing (NLP)](https://mlsanity.com/paper/2109.13123/) (2021)
- [Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard](https://mlsanity.com/paper/2305.18618/) (2023)

## Built on by (most cited first, 30 of 31)

- [IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory](https://mlsanity.com/paper/2506.01048/) (2025)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation](https://mlsanity.com/paper/2410.16547/) (2024)
- [Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction](https://mlsanity.com/paper/2512.18880/) (2025)
- [Do LLMs Agree on the Creativity Evaluation of Alternative Uses?](https://mlsanity.com/paper/2411.15560/) (2024)
- [Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study](https://mlsanity.com/paper/2508.08314/) (2025)
- [Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations](https://mlsanity.com/paper/2601.09953/) (2026)
- [Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators](https://mlsanity.com/paper/2507.05890/) (2025)
- [Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?](https://mlsanity.com/paper/2507.08232/) (2025)
- [LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment](https://mlsanity.com/paper/2606.18709/) (2026)
- [Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores](https://mlsanity.com/paper/2510.14966/) (2025)
- [Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation](https://mlsanity.com/paper/2606.15643/) (2026)
- [Automatic Generation of Inference Making Questions for Reading Comprehension Assessments](https://mlsanity.com/paper/2506.08260/) (2025)
- [The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulation](https://mlsanity.com/paper/2511.05903/) (2025)
- [Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.02580/) (2026)
- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment](https://mlsanity.com/paper/2605.19529/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [A Novel Psychometrics-Based Approach to Developing Professional Competency Benchmark for Large Language Models](https://mlsanity.com/paper/2411.00045/) (2024)
- [The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory](https://mlsanity.com/paper/2503.10533/) (2025)
- [Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI-Generated and Human Data](https://mlsanity.com/paper/2505.21997/) (2025)
- [Delving Into the Psychology of Machines: Exploring the Structure of Self-Regulated Learning via LLM-Generated Survey Responses](https://mlsanity.com/paper/2506.13384/) (2025)
- [Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests](https://mlsanity.com/paper/2510.22170/) (2025)
- [Exploring the Psychometric Validity of AI-Generated Student Responses: A Study on Virtual Personas' Learning Motivation](https://mlsanity.com/paper/2511.07451/) (2025)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling](https://mlsanity.com/paper/2608.04554/) (2026)
- [Natural Language Processing Psychometrics](https://mlsanity.com/paper/2608.07316/) (2026)
- [Multimodal Item Parameter Estimation using Simulated Response Probabilitie](https://mlsanity.com/paper/2608.10154/) (2026)
- [A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications](https://mlsanity.com/paper/2608.16893/) (2026)
- [Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses](https://mlsanity.com/paper/2608.17810/) (2026)

## Abstract (arXiv)

Effective educational measurement relies heavily on the curation of well-designed item pools (i.e., possessing the right psychometric properties). However, item calibration is time-consuming and costly, requiring a sufficient number of respondents for the response process. We explore using six different LLMs (GPT-3.5, GPT-4, Llama 2, Llama 3, Gemini-Pro, and Cohere Command R Plus) and various combinations of them using sampling methods to produce responses with psychometric properties similar to human answers. Results show that some LLMs have comparable or higher proficiency in College Algebra than college students. No single LLM mimics human respondents due to narrow proficiency distributions, but an ensemble of LLMs can better resemble college students' ability distribution. The item parameters calibrated by LLM-Respondents have high correlations (e.g. > 0.8 for GPT-3.5) compared to their human calibrated counterparts, and closely resemble the parameters of the human subset (e.g. 0.02 Spearman correlation difference). Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).