mlsanityData to 7 Oct 2026

Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

arXiv 2405.19132 ↗First posted 29 May 2024cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 26 of its 58 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

26 of 58 references.

References · 26

  1. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  2. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  3. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  4. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 85
  5. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  6. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 81
  7. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 71
  8. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  9. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 63
  10. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
  11. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  12. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 31
14 more
  1. 2022Repairing Bugs in Python Assignments Using Large Language Modelscited by 28
  2. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  3. 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
  4. 2023Transformative Effects of ChatGPT on Modern Education: Emerging Era of AI Chatbotscited by 18
  5. 2023Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generatorscited by 16
  6. 2023Trust in Generative AI among students: An Exploratory Studycited by 16
  7. 2023Can Language Models Employ the Socratic Method? Experiments with Code Debuggingcited by 14
  8. 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
  9. 2023ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questionscited by 13
  10. 2023Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Modelscited by 11
  11. 2023Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessmentscited by 10
  12. 2023Can ChatGPT Pass An Introductory Level Functional Language Programming Course?cited by 9
  13. 2023AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?cited by 8
  14. 2024Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethicscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  2. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 13
  3. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 12
  4. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
  5. 2025The Role of Generative AI in Software Student CollaborAItioncited by 2
  6. 2025Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?cited by 1
  7. 2025GenAI Voice Mode in Programming Educationcited by 0
  8. 2026AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspectivecited by 0