# Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?

arXiv: https://arxiv.org/abs/2507.08232 · HTML: https://mlsanity.com/paper/2507.08232/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks](https://mlsanity.com/paper/2408.08631/) (2024)
- [Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models](https://mlsanity.com/paper/2407.15645/) (2024)
- [Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency](https://mlsanity.com/paper/2310.06837/) (2023)
- [LLM-based Cognitive Models of Students with Misconceptions](https://mlsanity.com/paper/2410.12294/) (2024)
- [AI Agents and Education: Simulated Practice at Scale](https://mlsanity.com/paper/2407.12796/) (2024)
- [Apprentice Tutor Builder: A Platform For Users to Create and Personalize Intelligent Tutors](https://mlsanity.com/paper/2404.07883/) (2024)

## Built on by (most cited first, 3 of 3)

- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation](https://mlsanity.com/paper/2608.21668/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used as proxy students in the development of Intelligent Tutoring Systems (ITSs) and in piloting test questions. However, to what extent these proxy students accurately emulate the behavior and characteristics of real students remains an open question. To investigate this, we collected a dataset of 489 items from the National Assessment of Educational Progress (NAEP), covering mathematics and reading comprehension in grades 4, 8, and 12. We then apply an Item Response Theory (IRT) model to position 11 diverse and state-of-the-art LLMs on the same ability scale as real student populations. Our findings reveal that, without guidance, strong general-purpose models consistently outperform the average student at every grade, while weaker or domain-mismatched models may align incidentally. Using grade-enforcement prompts changes models' performance, but whether they align with the average grade-level student remains highly model- and prompt-specific: no evaluated model-prompt pair fits the bill across subjects and grades, underscoring the need for new training and evaluation strategies. We conclude by providing guidelines for the selection of viable proxies based on our findings.