# The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

arXiv: https://arxiv.org/abs/2405.17739 · HTML: https://mlsanity.com/paper/2405.17739/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 24 of 24)

- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [The Effects of Generative AI on Computing Students' Help-Seeking Preferences](https://mlsanity.com/paper/2401.02262/) (2024)
- [Generative AI in Computing Education: Perspectives of Students and Instructors](https://mlsanity.com/paper/2308.04309/) (2023)
- [Desirable Characteristics for AI Teaching Assistants in Programming Education](https://mlsanity.com/paper/2405.14178/) (2024)
- [CS1-LLM: Integrating LLMs into CS1 Instruction](https://mlsanity.com/paper/2406.15379/) (2024)
- [Ivie: Lightweight Anchored Explanations of Just-Generated Code](https://mlsanity.com/paper/2403.02491/) (2024)
- [More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems](https://mlsanity.com/paper/2311.04926/) (2023)
- ["ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses](https://mlsanity.com/paper/2404.17443/) (2024)

## Built on by (most cited first, 30 of 48)

- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [How Do Programming Students Use Generative AI?](https://mlsanity.com/paper/2501.10091/) (2025)
- [Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaboration](https://mlsanity.com/paper/2502.11882/) (2025)
- [Beyond Text-to-Text: An Overview of Multimodal and Generative Artificial Intelligence for Education Using Topic Modeling](https://mlsanity.com/paper/2409.16376/) (2024)
- [Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing Courses](https://mlsanity.com/paper/2410.06865/) (2024)
- [Generative AI in Education: From Foundational Insights to the Socratic Playground for Learning](https://mlsanity.com/paper/2501.06682/) (2025)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [The Evolving Usage of GenAI by Computing Students](https://mlsanity.com/paper/2412.16453/) (2024)
- [Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education](https://mlsanity.com/paper/2502.20527/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails](https://mlsanity.com/paper/2504.11146/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models](https://mlsanity.com/paper/2412.11088/) (2024)
- [Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners](https://mlsanity.com/paper/2412.12800/) (2024)
- [Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models](https://mlsanity.com/paper/2503.04267/) (2025)
- [Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education](https://mlsanity.com/paper/2506.19107/) (2025)
- [Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Education](https://mlsanity.com/paper/2512.23982/) (2025)
- ["Bespoke Bots": Diverse Instructor Needs for Customizing Generative AI Classroom Chatbots](https://mlsanity.com/paper/2603.00057/) (2026)
- [Unpacking Vibe Coding: Help-Seeking Processes in Student-AI Interactions While Programming](https://mlsanity.com/paper/2604.27134/) (2026)
- [Integrating Natural Language Prompting Tasks in Introductory Programming Courses](https://mlsanity.com/paper/2410.03063/) (2024)
- ["All Roads Lead to ChatGPT": How Generative AI is Eroding Social Interactions and Student Learning Communities](https://mlsanity.com/paper/2504.09779/) (2025)
- [User Misconceptions of LLM-Based Conversational Programming Assistants](https://mlsanity.com/paper/2510.25662/) (2025)
- [LLMs Integration in Software Engineering Team Projects: Roles, Impact, and a Pedagogical Design Space for AI Tools in Computing Education](https://mlsanity.com/paper/2410.23069/) (2024)
- [Towards Pedagogical LLMs with Supervised Fine Tuning for Computing Education](https://mlsanity.com/paper/2411.01765/) (2024)
- [To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes](https://mlsanity.com/paper/2501.11935/) (2025)
- [Can AI support student engagement in classroom activities in higher education?](https://mlsanity.com/paper/2506.18941/) (2025)
- [GenAI Voice Mode in Programming Education](https://mlsanity.com/paper/2509.10596/) (2025)

## Abstract (arXiv)

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.