mlsanityData to 14 Oct 2026

Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments

arXiv 2308.08572 ↗First posted 15 Aug 2023cs.SE cs.AI cs.HCcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper investigates the performance of the Large Language Models (LLMs) ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the performance, implications for didactic scenarios and assessment formats utilizing LLMs are derived. For the analysis, 72 Python tasks for novice programmers were selected from the free site CodingBat. Full task descriptions were used as input to the LLMs, while the generated replies were evaluated using CodingBat's unit tests. In addition, the general availability of textual explanations and program code was analyzed. The results show high scores of 94.4 to 95.8% correct responses and reliable availability of textual explanations and program code, which opens new ways to incorporate LLMs into programming education and assessment.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 7 of its 28 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

7 of 28 references.

References · 7

  1. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  2. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
  3. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
  4. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  5. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
  6. 2022Repairing Bugs in Python Assignments Using Large Language Modelscited by 28
  7. 2020Zur Modellierung und Klassifizierung von Kompetenzen in der grundlegenden Programmierausbildung anhand der Anderson Krathwohl Taxonomiecited by 1

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 6; 2025: 1.

10 papers.

  1. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  2. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  3. 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
  4. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
  5. 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
  6. 2024Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTcited by 8
  7. 2024You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?cited by 6
  8. 2023Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligencecited by 4
  9. 2025Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?cited by 1
  10. 2024Scaling CS1 Support with Compiler-Integrated Conversational AIcited by 0