mlsanityData to 14 Oct 2026

Evaluating LLM-Generated Q&A Test: a Student-Centered Study

arXiv 2505.06591 ↗First posted 10 May 2025cs.CL cs.HCcited by 3

Not on the map. Why?

Abstract (arXiv)

This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o-mini-based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 3 of its 14 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

3 of 14 references.

References · 3

  1. 2024Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluationcited by 29
  2. 2024An Automatic Question Usability Evaluation Toolkitcited by 10
  3. 2025Applying Text Mining to Analyze Human Question Asking in Creativity Researchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Modelscited by 0
  2. 2026From Surface Learning to Deep Understanding: A Grounded AI Tutoring System for Moodlecited by 0
  3. 2026A Benchmark for LLM's Understanding of Middle School and High School Science Topicscited by 0