# ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions

arXiv: https://arxiv.org/abs/2304.14993 · HTML: https://mlsanity.com/paper/2304.14993/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 7 of 7)

- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?](https://mlsanity.com/paper/2303.11146/) (2023)

## Built on by (most cited first, 13 of 13)

- [Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics](https://mlsanity.com/paper/2503.23989/) (2025)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Beyond Black Box AI-Generated Plagiarism Detection: From Sentence to Document Level](https://mlsanity.com/paper/2306.08122/) (2023)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review](https://mlsanity.com/paper/2503.00079/) (2025)
- [ChatGPT in Data Visualization Education: A Student Perspective](https://mlsanity.com/paper/2405.00748/) (2024)
- [BoilerTAI: A Platform for Enhancing Instruction Using Generative AI in Educational Forums](https://mlsanity.com/paper/2409.13196/) (2024)
- [Generative AI Takes a Statistics Exam: A Comparison of Performance between ChatGPT3.5, ChatGPT4, and ChatGPT4o-mini](https://mlsanity.com/paper/2501.09171/) (2025)
- [Assessing the Capability of LLMs in Solving POSCOMP Questions](https://mlsanity.com/paper/2505.20338/) (2025)
- ["Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students](https://mlsanity.com/paper/2402.01687/) (2024)
- [Utilizing ChatGPT in a Data Structures and Algorithms Course: A Teaching Assistant's Perspective](https://mlsanity.com/paper/2410.08899/) (2024)
- [Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026](https://mlsanity.com/paper/2608.16318/) (2026)

## Abstract (arXiv)

ChatGPT is an AI language model developed by OpenAI that can understand and generate human-like text. It can be used for a variety of use cases such as language generation, question answering, text summarization, chatbot development, language translation, sentiment analysis, content creation, personalization, text completion, and storytelling. While ChatGPT has garnered significant positive attention, it has also generated a sense of apprehension and uncertainty in academic circles. There is concern that students may leverage ChatGPT to complete take-home assignments and exams and obtain favorable grades without genuinely acquiring knowledge. This paper adopts a quantitative approach to demonstrate ChatGPT's high degree of unreliability in answering a diverse range of questions pertaining to topics in undergraduate computer science. Our analysis shows that students may risk self-sabotage by blindly depending on ChatGPT to complete assignments and exams. We build upon this analysis to provide constructive recommendations to both students and instructors.