# LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI

arXiv: https://arxiv.org/abs/2503.18995 · HTML: https://mlsanity.com/paper/2503.18995/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 12 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Builds on (most cited first, 2 of 2)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks](https://mlsanity.com/paper/2407.13511/) (2024)

## Built on by (most cited first, 3 of 3)

- [Enhancing Student Learning with LLM-Generated Retrieval Practice Questions: An Empirical Study in Data Science Courses](https://mlsanity.com/paper/2507.05629/) (2025)
- [AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study](https://mlsanity.com/paper/2504.08846/) (2025)
- [Rate-Distortion Guided Knowledge Graph Construction from Lecture Notes Using Gromov-Wasserstein Optimal Transport](https://mlsanity.com/paper/2511.14595/) (2025)

## Abstract (arXiv)

We randomly deploy questions constructed with and without use of the LLM tool and gauge the ability of the students to correctly answer, as well as their ability to correctly perceive the difference between human-authored and LLM-authored questions. In determining whether the questions written with the aid of ChatGPT were consistent with the instructor's questions and source text, we computed representative vectors of both the human and ChatGPT questions using SBERT and compared cosine similarity to the course textbook. A non-significant Mann-Whitney U test (z = 1.018, p = .309) suggests that students were unable to perceive whether questions were written with or without the aid of ChatGPT. However, student scores on LLM-authored questions were almost 9% lower (z = 2.702, p < .01). This result may indicate that either the AI questions were more difficult or that the students were more familiar with the instructor's style of questions. Overall, the study suggests that while there is potential for using LLM tools to aid in the construction of assessments, care must be taken to ensure that the questions are fair, well-composed, and relevant to the course material.