# Evaluating LLM-Generated Q&A Test: a Student-Centered Study

arXiv: https://arxiv.org/abs/2505.06591 · HTML: https://mlsanity.com/paper/2505.06591/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 14 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 3 of 3)

- [Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluation](https://mlsanity.com/paper/2408.04394/) (2024)
- [An Automatic Question Usability Evaluation Toolkit](https://mlsanity.com/paper/2405.20529/) (2024)
- [Applying Text Mining to Analyze Human Question Asking in Creativity Research](https://mlsanity.com/paper/2501.02090/) (2025)

## Built on by (most cited first, 3 of 3)

- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)
- [From Surface Learning to Deep Understanding: A Grounded AI Tutoring System for Moodle](https://mlsanity.com/paper/2605.06963/) (2026)
- [A Benchmark for LLM's Understanding of Middle School and High School Science Topics](https://mlsanity.com/paper/2609.32020/) (2026)

## Abstract (arXiv)

This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o-mini-based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.