# Student Mastery or AI Deception? Analyzing ChatGPT's Assessment Proficiency and Evaluating Detection Strategies

arXiv: https://arxiv.org/abs/2311.16292 · HTML: https://mlsanity.com/paper/2311.16292/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?](https://mlsanity.com/paper/2303.11146/) (2023)
- [Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases](https://mlsanity.com/paper/2307.07411/) (2023)
- [ChatGPT Participates in a Computer Science Exam](https://mlsanity.com/paper/2303.09461/) (2023)
- [Can ChatGPT Pass An Introductory Level Functional Language Programming Course?](https://mlsanity.com/paper/2305.02230/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Knowledge-Component-Based Methodology for Evaluating AI Assistants](https://mlsanity.com/paper/2406.05603/) (2024)

## Abstract (arXiv)

Generative AI systems such as ChatGPT have a disruptive effect on learning and assessment. Computer science requires practice to develop skills in problem solving and programming that are traditionally developed using assignments. Generative AI has the capability of completing these assignments for students with high accuracy, which dramatically increases the potential for academic integrity issues and students not achieving desired learning outcomes. This work investigates the performance of ChatGPT by evaluating it across three courses (CS1,CS2,databases). ChatGPT completes almost all introductory assessments perfectly. Existing detection methods, such as MOSS and JPlag (based on similarity metrics) and GPTzero (AI detection), have mixed success in identifying AI solutions. Evaluating instructors and teaching assistants using heuristics to distinguish between student and AI code shows that their detection is not sufficiently accurate. These observations emphasize the need for adapting assessments and improved detection methods.