# How Beginning Programmers and Code LLMs (Mis)read Each Other

arXiv: https://arxiv.org/abs/2401.15232 · HTML: https://mlsanity.com/paper/2401.15232/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2024.

## Built on by (most cited first, 30 of 31)

- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use](https://mlsanity.com/paper/2409.08775/) (2024)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [API-guided Dataset Synthesis to Finetune Large Code Models](https://mlsanity.com/paper/2408.08343/) (2024)
- [Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches](https://mlsanity.com/paper/2405.03998/) (2024)
- [The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired](https://mlsanity.com/paper/2503.16491/) (2025)
- [Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs](https://mlsanity.com/paper/2506.17353/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice](https://mlsanity.com/paper/2409.18661/) (2024)
- [Assessing LLM code generation quality through path planning tasks](https://mlsanity.com/paper/2504.21276/) (2025)
- [Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education](https://mlsanity.com/paper/2506.19107/) (2025)
- [More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processes](https://mlsanity.com/paper/2603.24877/) (2026)
- [DeepCodeProbe: Towards Understanding What Models Trained on Code Learn](https://mlsanity.com/paper/2407.08890/) (2024)
- [Integrating Natural Language Prompting Tasks in Introductory Programming Courses](https://mlsanity.com/paper/2410.03063/) (2024)
- [What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Code](https://mlsanity.com/paper/2412.08098/) (2024)
- [Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning](https://mlsanity.com/paper/2501.15727/) (2025)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)
- [User Misconceptions of LLM-Based Conversational Programming Assistants](https://mlsanity.com/paper/2510.25662/) (2025)
- [To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes](https://mlsanity.com/paper/2501.11935/) (2025)
- [A Showdown of ChatGPT vs DeepSeek in Solving Programming Tasks](https://mlsanity.com/paper/2503.13549/) (2025)
- [A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case](https://mlsanity.com/paper/2510.21933/) (2025)
- [Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming](https://mlsanity.com/paper/2511.13271/) (2025)
- [Feasibility of AI-Assisted Programming for End-User Development](https://mlsanity.com/paper/2512.05666/) (2025)
- [Assessing the Software Security Comprehension of Large Language Models](https://mlsanity.com/paper/2512.21238/) (2025)
- [PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions](https://mlsanity.com/paper/2601.15575/) (2026)

## Abstract (arXiv)

Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education.