# How Do Programming Students Use Generative AI? arXiv: https://arxiv.org/abs/2501.10091 ยท HTML: https://mlsanity.com/paper/2501.10091/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 52 references that are in the map. - First posted on arXiv 17 Jan 2025. ## Builds on (most cited first, 23 of 23) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023) - [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023) - [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022) - [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021) - [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022) - [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023) - [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023) - [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022) - [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022) - ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023) - [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023) - [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024) - [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023) - [Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions](https://mlsanity.com/paper/2308.02312/) (2023) - [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022) - [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023) - [Generative AI in Computing Education: Perspectives of Students and Instructors](https://mlsanity.com/paper/2308.04309/) (2023) - [CS1-LLM: Integrating LLMs into CS1 Instruction](https://mlsanity.com/paper/2406.15379/) (2024) - [Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2402.11702/) (2024) - [Can Developers Prompt? A Controlled Experiment for Code Documentation Generation](https://mlsanity.com/paper/2408.00686/) (2024) - [A Simple NLP-based Approach to Support Onboarding and Retention in Open Source Communities](https://mlsanity.com/paper/1806.02592/) (2018) - [ChatGPT in the classroom. Exploring its potential and limitations in a Functional Programming course](https://mlsanity.com/paper/2401.11166/) (2024) ## Built on by (most cited first, 10 of 10) - [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025) - [Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks](https://mlsanity.com/paper/2508.04125/) (2025) - [Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming](https://mlsanity.com/paper/2511.13271/) (2025) - [From Pilots to Practices: A Scoping Review of GenAI-Enabled Personalization in Computer Science Education](https://mlsanity.com/paper/2512.20714/) (2025) - [Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems](https://mlsanity.com/paper/2604.07304/) (2026) - [From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data](https://mlsanity.com/paper/2607.16845/) (2026) - [How Well Can AI Generate Backlogs from App Mockups?](https://mlsanity.com/paper/2607.22902/) (2026) - [Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course](https://mlsanity.com/paper/2607.24755/) (2026) - [Do Not Copy/Paste: Soft Barriers for Copying in AI-Assisted Programming](https://mlsanity.com/paper/2608.22638/) (2026) - [Understanding Generative AI Use in Programming MOOCs: The Role of Course Context and Learner Characteristics](https://mlsanity.com/paper/2610.04447/) (2026) ## Abstract (arXiv) Programming students have a widespread access to powerful Generative AI tools like ChatGPT. While this can help understand the learning material and assist with exercises, educators are voicing more and more concerns about an overreliance on generated outputs and lack of critical thinking skills. It is thus important to understand how students actually use generative AI and what impact this could have on their learning behavior. To this end, we conducted a study including an exploratory experiment with 37 programming students, giving them monitored access to ChatGPT while solving a code authoring exercise. The task was not directly solvable by ChatGPT and required code comprehension and reasoning. While only 23 of the students actually opted to use the chatbot, the majority of those eventually prompted it to simply generate a full solution. We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions. Instead of using the bot to comprehend the code and their own mistakes, students often got trapped in a vicious cycle of submitting wrong generated code and then asking the bot for a fix. Those who self-reported using generative AI regularly were more likely to prompt the bot to generate a solution. Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified. We discuss how researchers and educators can respond to the potential risk of students uncritically over-relying on Generative AI. We also discuss potential modifications to our study design for large-scale replications.