# How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment

arXiv: https://arxiv.org/abs/2309.14049 · HTML: https://mlsanity.com/paper/2309.14049/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2023.

## Built on by (most cited first, 30 of 56)

- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents](https://mlsanity.com/paper/2406.06613/) (2024)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support](https://mlsanity.com/paper/2502.18658/) (2025)
- [The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers](https://mlsanity.com/paper/2404.02806/) (2024)
- [An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project](https://mlsanity.com/paper/2401.16186/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices](https://mlsanity.com/paper/2404.02213/) (2024)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code](https://mlsanity.com/paper/2502.18851/) (2025)
- [Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving](https://mlsanity.com/paper/2503.16434/) (2025)
- [Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course](https://mlsanity.com/paper/2406.06451/) (2024)
- [AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instruct](https://mlsanity.com/paper/2405.14906/) (2024)
- [CourseAssist: Pedagogically Appropriate AI Tutor for Computer Science Education](https://mlsanity.com/paper/2407.10246/) (2024)
- [TableTalk: Scaffolding Spreadsheet Development with a Language Agent](https://mlsanity.com/paper/2502.09787/) (2025)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [Using Large Language Models for education managements in Vietnamese with low resources](https://mlsanity.com/paper/2501.15022/) (2025)
- [Rethinking the Outlier Distribution in Large Language Models: An In-depth Study](https://mlsanity.com/paper/2505.21670/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [ChatGPT in Data Visualization Education: A Student Perspective](https://mlsanity.com/paper/2405.00748/) (2024)
- [Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs](https://mlsanity.com/paper/2405.13932/) (2024)
- [Envisioning the Next-Generation AI Coding Assistants: Insights & Proposals](https://mlsanity.com/paper/2403.14592/) (2024)
- [Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning](https://mlsanity.com/paper/2404.18262/) (2024)

## Abstract (arXiv)

As Large Language Models (LLMs) gain in popularity, it is important to understand how novice programmers use them. We present a thematic analysis of 33 learners, aged 10-17, independently learning Python through 45 code-authoring tasks using Codex, an LLM-based code generator. We explore several questions related to how learners used these code generators and provide an analysis of the properties of the written prompts and the generated code. Specifically, we explore (A) the context in which learners use Codex, (B) what learners are asking from Codex, (C) properties of their prompts in terms of relation to task description, language, and clarity, and prompt crafting patterns, (D) the correctness, complexity, and accuracy of the AI-generated code, and (E) how learners utilize AI-generated code in terms of placement, verification, and manual modifications. Furthermore, our analysis reveals four distinct coding approaches when writing code with an AI code generator: AI Single Prompt, where learners prompted Codex once to generate the entire solution to a task; AI Step-by-Step, where learners divided the problem into parts and used Codex to generate each part; Hybrid, where learners wrote some of the code themselves and used Codex to generate others; and Manual coding, where learners wrote the code themselves. The AI Single Prompt approach resulted in the highest correctness scores on code-authoring tasks, but the lowest correctness scores on subsequent code-modification tasks during training. Our results provide initial insight into how novice learners use AI code generators and the challenges and opportunities associated with integrating them into self-paced learning environments. We conclude with various signs of over-reliance and self-regulation, as well as opportunities for curriculum and tool development.