# Substance Beats Style: Why Beginning Students Fail to Code with LLMs

arXiv: https://arxiv.org/abs/2410.19792 · HTML: https://mlsanity.com/paper/2410.19792/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM](https://mlsanity.com/paper/2403.19114/) (2024)
- [What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use](https://mlsanity.com/paper/2409.08775/) (2024)
- [GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agency](https://mlsanity.com/paper/2402.08855/) (2024)
- [Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators](https://mlsanity.com/paper/2307.16364/) (2023)
- [Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?](https://mlsanity.com/paper/2210.14699/) (2022)
- [Keyframer: Empowering Animation Design using Large Language Models](https://mlsanity.com/paper/2402.06071/) (2024)
- [Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks](https://mlsanity.com/paper/2410.01026/) (2024)
- [Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches](https://mlsanity.com/paper/2405.03998/) (2024)
- [EditScribe: Non-Visual Image Editing with Natural Language Verification Loops](https://mlsanity.com/paper/2408.06632/) (2024)
- [NoviCode: Generating Programs from Natural Language Utterances by Novices](https://mlsanity.com/paper/2407.10626/) (2024)
- [Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Models](https://mlsanity.com/paper/2401.10759/) (2024)

## Built on by (most cited first, 5 of 5)

- [Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025](https://mlsanity.com/paper/2512.14012/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Education](https://mlsanity.com/paper/2512.23982/) (2025)
- [Educating the Agentic Engineer: Curricula, Collaboration, and Continuous Learning in the AI Era](https://mlsanity.com/paper/2607.29610/) (2026)

## Abstract (arXiv)

Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks. Why is this the case? This paper explores two competing hypotheses about the cause of student-LLM miscommunication: (1) students simply lack the technical vocabulary needed to write good prompts, and (2) students do not understand the extent of information that LLMs need to solve code generation tasks. We study (1) with a causal intervention experiment on technical vocabulary and (2) by analyzing graphs that abstract how students edit prompts and the different failures that they encounter. We find that substance beats style: a poor grasp of technical vocabulary is merely correlated with prompt failure; that the information content of prompts predicts success; that students get stuck making trivial edits; and more. Our findings have implications for the use of LLMs in programming education, and for efforts to make computing more accessible with LLMs.