# Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for Families

arXiv: https://arxiv.org/abs/2305.10417 · HTML: https://mlsanity.com/paper/2305.10417/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 11 of 11)

- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement](https://mlsanity.com/paper/2202.06205/) (2022)
- [Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias](https://mlsanity.com/paper/2212.10678/) (2022)
- [WhyGen: Explaining ML-powered Code Generation by Referring to Training Examples](https://mlsanity.com/paper/2204.07940/) (2022)
- [A Study of Editor Features in a Creative Coding Classroom](https://mlsanity.com/paper/2301.13302/) (2023)

## Built on by (most cited first, 5 of 5)

- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Can Large Language Models Find And Fix Vulnerable Software?](https://mlsanity.com/paper/2308.10345/) (2023)
- [AI-Assisted Causal Pathway Diagram for Human-Centered Design](https://mlsanity.com/paper/2403.08111/) (2024)
- [Scratch Copilot: Supporting Youth Creative Coding with AI](https://mlsanity.com/paper/2505.03867/) (2025)
- [DesignerlyLoop: Forming Design Intent through Curated Reasoning for Human-LLM Alignment](https://mlsanity.com/paper/2511.15331/) (2025)

## Abstract (arXiv)

How can AI enhance creative coding experiences for families? This study explores the potential of large language models (LLMs) in helping families with creative coding using Scratch. Based on our previous user study involving a prototype AI assistant, we devised three evaluation scenarios to determine if LLMs could help families comprehend game code, debug programs, and generate new ideas for future projects. We utilized 22 Scratch projects for each scenario and generated responses from LLMs with and without practice tasks, resulting in 120 creative coding support scenario datasets. In addition, the authors independently evaluated their precision, pedagogical value, and age-appropriate language. Our findings show that LLMs achieved an overall success rate of more than 80% on the different tasks and evaluation criteria. This research offers valuable information on using LLMs for creative family coding and presents design guidelines for future AI-supported coding applications. Our evaluation framework, together with our labeled evaluation data, is publicly available.