# QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulation

arXiv: https://arxiv.org/abs/2503.05888 · HTML: https://mlsanity.com/paper/2503.05888/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Simulating Classroom Education with LLM-Empowered Agents](https://mlsanity.com/paper/2406.19226/) (2024)
- [Branch-Solve-Merge Improves Large Language Model Evaluation and Generation](https://mlsanity.com/paper/2310.15123/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation](https://mlsanity.com/paper/2402.05699/) (2024)
- [Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds](https://mlsanity.com/paper/1908.11421/) (2019)
- [CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent](https://mlsanity.com/paper/2310.06155/) (2023)
- [Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs](https://mlsanity.com/paper/2205.00355/) (2022)
- [Leveraging generative artificial intelligence to simulate student learning behavior](https://mlsanity.com/paper/2310.19206/) (2023)
- [EQG-RACE: Examination-Type Question Generation](https://mlsanity.com/paper/2012.06106/) (2020)
- [Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Study](https://mlsanity.com/paper/2406.11629/) (2024)
- [Evaluating the Knowledge Dependency of Questions](https://mlsanity.com/paper/2211.11902/) (2022)
- [Which questions should I answer? Salience Prediction of Inquisitive Questions](https://mlsanity.com/paper/2404.10917/) (2024)
- [Reference-based Metrics Disprove Themselves in Question Generation](https://mlsanity.com/paper/2403.12242/) (2024)

## Built on by (most cited first, 3 of 3)

- [Automated Benchmark Generation from Domain Guidelines Informed by Bloom's Taxonomy](https://mlsanity.com/paper/2601.20253/) (2026)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [From Learner Behavior to Reusable Skills for Effective and Efficient Learner Simulation](https://mlsanity.com/paper/2609.37157/) (2026)

## Abstract (arXiv)

While the Question Generation (QG) task has been increasingly adopted in educational assessments, its evaluation remains limited by approaches that lack a clear connection to the educational values of test items. In this work, we introduce test item analysis, a method frequently used by educators to assess test question quality, into QG evaluation. Specifically, we construct pairs of candidate questions that differ in quality across dimensions such as topic coverage, item difficulty, item discrimination, and distractor efficiency. We then examine whether existing QG evaluation approaches can effectively distinguish these differences. Our findings reveal significant shortcomings in these approaches with respect to accurately assessing test item quality in relation to student performance. To address this gap, we propose a novel QG evaluation framework, QG-SMS, which leverages Large Language Model for Student Modeling and Simulation to perform test item analysis. As demonstrated in our extensive experiments and human evaluation study, the additional perspectives introduced by the simulated student profiles lead to a more effective and robust assessment of test items.