# Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments

arXiv: https://arxiv.org/abs/2412.02713 · HTML: https://mlsanity.com/paper/2412.02713/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 32 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 4 of 4)

- [ChatGPT: The End of Online Exam Integrity?](https://mlsanity.com/paper/2212.09292/) (2022)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving](https://mlsanity.com/paper/2310.08773/) (2023)
- [Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants](https://mlsanity.com/paper/2408.11841/) (2024)

## Built on by (most cited first, 3 of 3)

- [Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots](https://mlsanity.com/paper/2603.23682/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses](https://mlsanity.com/paper/2608.17810/) (2026)

## Abstract (arXiv)

Generative AI is transforming the educational landscape, raising significant concerns about cheating. Despite the widespread use of multiple-choice questions in assessments, the detection of AI cheating in MCQ-based tests has been almost unexplored, in contrast to the focus on detecting AI-cheating on text-rich student outputs. In this paper, we propose a method based on the application of Item Response Theory to address this gap. Our approach operates on the assumption that artificial and human intelligence exhibit different response patterns, with AI cheating manifesting as deviations from the expected patterns of human responses. These deviations are modeled using Person-Fit Statistics. We demonstrate that this method effectively highlights the differences between human responses and those generated by premium versions of leading chatbots (ChatGPT, Claude, and Gemini), but that it is also sensitive to the amount of AI cheating in the data. Furthermore, we show that the chatbots differ in their reasoning profiles. Our work provides both a theoretical foundation and empirical evidence for the application of IRT to identify AI cheating in MCQ-based assessments.