# How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course

arXiv: https://arxiv.org/abs/2407.20792 · HTML: https://mlsanity.com/paper/2407.20792/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Repairing Bugs in Python Assignments Using Large Language Models](https://mlsanity.com/paper/2209.14876/) (2022)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Next-Step Hint Generation for Introductory Programming Using Large Language Models](https://mlsanity.com/paper/2312.10055/) (2023)
- [Transformative Effects of ChatGPT on Modern Education: Emerging Era of AI Chatbots](https://mlsanity.com/paper/2306.03823/) (2023)
- [Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators](https://mlsanity.com/paper/2307.16364/) (2023)
- [Generative AI in Computing Education: Perspectives of Students and Instructors](https://mlsanity.com/paper/2308.04309/) (2023)
- [Trust in Generative AI among students: An Exploratory Study](https://mlsanity.com/paper/2310.04631/) (2023)
- [Can Language Models Employ the Socratic Method? Experiments with Code Debugging](https://mlsanity.com/paper/2310.03210/) (2023)
- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions](https://mlsanity.com/paper/2304.14993/) (2023)
- [Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models](https://mlsanity.com/paper/2308.11873/) (2023)
- [Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments](https://mlsanity.com/paper/2308.08572/) (2023)
- [Can ChatGPT Pass An Introductory Level Functional Language Programming Course?](https://mlsanity.com/paper/2305.02230/) (2023)
- [AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?](https://mlsanity.com/paper/2311.10737/) (2023)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Generative AI Perceptions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia](https://mlsanity.com/paper/2304.14415/) (2023)
- [Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics](https://mlsanity.com/paper/2406.11858/) (2024)

## Built on by (most cited first, 11 of 11)

- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation](https://mlsanity.com/paper/2510.08996/) (2025)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?](https://mlsanity.com/paper/2412.03516/) (2024)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Do AI models help produce verified bug fixes?](https://mlsanity.com/paper/2507.15822/) (2025)
- [Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?](https://mlsanity.com/paper/2507.17258/) (2025)
- [GenAI Voice Mode in Programming Education](https://mlsanity.com/paper/2509.10596/) (2025)
- [AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective](https://mlsanity.com/paper/2606.22809/) (2026)
- [AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering](https://mlsanity.com/paper/2608.04148/) (2026)
- [Analysis of Types of Inquiries in Student-AI Interaction: A case study of two CS2 tasks](https://mlsanity.com/paper/2608.17919/) (2026)

## Abstract (arXiv)

This research paper contributes to the computing education research community's understanding of Generative AI (GenAI) in the context of introductory programming, and specifically, how students utilize related tools, such as ChatGPT. An increased understanding of students' use is mandatory for educators and higher education institutions, as GenAI is here to stay, and its performance is likely to improve rapidly in the near future. Learning about students' use patterns is not only crucial to support their learning, but to develop adequate forms of instruction and assessment. With the rapid advancement of AI, its broad availability, and ubiquitous presence in educational environments, elaborating how AI can enhance learning experiences, especially in courses such as introductory programming is important. To date, most studies have focused on the educator's perspective on GenAI, its performance, characteristics, and limitations. However, the student perspective, and how they actually use GenAI tools in course contexts, has not been subject to a great number of studies. Therefore, this study is guided by the following research questions: (1) What do students report on their use pattern of ChatGPT in the context of introductory programming exercises? and (2) How do students perceive ChatGPT in the context of introductory programming exercises? To address these questions, computing students at a large German university were asked to solve programming tasks with the assistance of ChatGPT as part of their introductory programming course. Students (n=298) provided information regarding the use of ChatGPT, and their evaluation of the tool via an online survey. This research provides a comprehensive evaluation of ChatGPT-3.5's application by novice programmers in a higher education context...