mlsanityData to 7 Oct 2026

Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation

arXiv 2405.11591 ↗First posted 19 May 2024cs.HC cs.AIcited by 56

Not on the map. Why?

Abstract (arXiv)

Evaluating the quality of automatically generated question items has been a long standing challenge. In this paper, we leverage LLMs to simulate student profiles and generate responses to multiple-choice questions (MCQs). The generative students' responses to MCQs can further support question item evaluation. We propose Generative Students, a prompt architecture designed based on the KLI framework. A generative student profile is a function of the list of knowledge components the student has mastered, has confusion about or has no evidence of knowledge of. We instantiate the Generative Students concept on the subject domain of heuristic evaluation. We created 45 generative students using GPT-4 and had them respond to 20 MCQs. We found that the generative students produced logical and believable responses that were aligned with their profiles. We then compared the generative students' responses to real students' responses on the same set of MCQs and found a high correlation. Moreover, there was considerable overlap in the difficult questions identified by generative students and real students. A subsequent case study demonstrated that an instructor could improve question quality based on the signals provided by Generative Students.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 13 of its 56 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

13 of 56 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  6. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  7. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  8. 2022Language models show human-like content effects on reasoning taskscited by 171
  9. 2023Rehearsal: Simulating Conflict to Teach Conflict Resolutioncited by 54
  10. 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 38
  11. 2023Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methodscited by 23
  12. 2023Leveraging generative artificial intelligence to simulate student learning behaviorcited by 16
1 more
  1. 2023Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineeringcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 26; 2026: 23.

56 papers.

Sort:

By area:

  1. 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
  2. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  3. 2024TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Studentscited by 25
  4. 2025EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Frameworkcited by 23
  5. 2025Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errorscited by 19
  6. 2024Psychometric Alignment: Capturing Human Knowledge Distributions via Language Modelscited by 18
  7. 2025Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulationcited by 15
  8. 2025Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?cited by 14
  9. 2026Simulated Students in Tutoring Dialogues: Substance or Illusion?cited by 14
  10. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  11. 2024PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creationcited by 9
  12. 2025Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative Systemcited by 9
38 more
  1. 2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Predictioncited by 7
  2. 2026Towards Valid Student Simulation with Large Language Modelscited by 7
  3. 2024Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Studentscited by 6
  4. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6
  5. 2025Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AIcited by 5
  6. 2026Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulationscited by 5
  7. 2025Bridging the AI Adoption Gap: Designing an Interactive Pedagogical Agent for Higher Education Instructorscited by 4
  8. 2025Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?cited by 4
  9. 2026PATS: Personality-Aware Teaching Strategies with Large Language Model Tutorscited by 4
  10. 2025QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulationcited by 3
  11. 2025Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agentscited by 3
  12. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  13. 2025Building Scaffolding Dialogue Data with LLM-Simulated Novicescited by 2
  14. 2026Who Am I? History-Aware Profiles for Student Simulation in Tutoring Dialoguescited by 2
  15. 2026Edu-Theater: A Data-Efficient Agent Framework for Scalable Learner Behavior Simulation through Staging Roll-Callcited by 2
  16. 2025Examining the Role of LLM-Driven Interactions on Attention and Cognitive Engagement in Virtual Classroomscited by 1
  17. 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studiescited by 1
  18. 2025Automatic Generation of Inference Making Questions for Reading Comprehension Assessmentscited by 1
  19. 2025The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulationcited by 1
  20. 2026DrawSim-PD: Simulating Student Science Drawings to Support NGSS-Aligned Teacher Diagnostic Reasoningcited by 1
  21. 2026BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulationcited by 1
  22. 2026Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Mostcited by 1
  23. 2026SWIM: Student Writing Simulation via Proficiency-Conditioned Generationcited by 1
  24. 2025Objective Metrics for Evaluating Large Language Models Using External Data Sourcescited by 0
  25. 2025Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Visioncited by 0
  26. 2025OnlineMate: An LLM-Based Multi-Agent Companion System for Cognitive Support in Online Learningcited by 0
  27. 2025QuizRank: Picking Images by Quizzing VLMscited by 0
  28. 2025Harnessing LLM for Noise-Robust Cognitive Diagnosis in Web-Based Intelligent Education Systemscited by 0
  29. 2025Student Development Agent: Risk-free Simulation for Evaluating AIED Innovationscited by 0
  30. 2025DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Educationcited by 0
  31. 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answeringcited by 0
  32. 2026Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluationcited by 0
  33. 2026The RIGID Framework: Research-Integrated, Generative AI-Mediated Instructional Designcited by 0
  34. 2026A Multi-Agent Framework for Feature-Constrained Difficulty Control in Reading Comprehension Item Generationcited by 0
  35. 2026Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring Systemcited by 0
  36. 2026EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibrationcited by 0
  37. 2026AI Assistants Overassistcited by 0
  38. 2026Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approachcited by 0

50 most cited of 56 All 56 (JSON)