# Prompt Problems: A New Programming Exercise for the Generative AI Era

arXiv: https://arxiv.org/abs/2311.05943 · HTML: https://mlsanity.com/paper/2311.05943/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2023.

## Built on by (most cited first, 30 of 31)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers](https://mlsanity.com/paper/2311.10242/) (2023)
- [What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use](https://mlsanity.com/paper/2409.08775/) (2024)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [ACE: A LLM-based Negotiation Coaching System](https://mlsanity.com/paper/2410.01555/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors](https://mlsanity.com/paper/2403.15586/) (2024)
- ["ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses](https://mlsanity.com/paper/2404.17443/) (2024)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- [Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts](https://mlsanity.com/paper/2409.13449/) (2024)
- [AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review](https://mlsanity.com/paper/2503.00079/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [Enhancing Programming Error Messages in Real Time with Generative AI](https://mlsanity.com/paper/2402.08072/) (2024)
- [Generative AI and Power Imbalances in Global Education: Frameworks for Bias Mitigation](https://mlsanity.com/paper/2406.02966/) (2024)
- [Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice](https://mlsanity.com/paper/2409.18661/) (2024)
- [Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners](https://mlsanity.com/paper/2412.12800/) (2024)
- [The Role of Generative AI in Software Student CollaborAItion](https://mlsanity.com/paper/2501.14084/) (2025)
- [Integrating Natural Language Prompting Tasks in Introductory Programming Courses](https://mlsanity.com/paper/2410.03063/) (2024)
- [Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning](https://mlsanity.com/paper/2501.15727/) (2025)
- [From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering](https://mlsanity.com/paper/2602.00496/) (2026)
- [Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI](https://mlsanity.com/paper/2606.05983/) (2026)
- [Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning](https://mlsanity.com/paper/2510.05417/) (2025)
- [Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students' Independent LLM Use for Programming](https://mlsanity.com/paper/2607.03303/) (2026)
- [Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications](https://mlsanity.com/paper/2607.10674/) (2026)
- [Beyond Perspectives: A Trio-Ethnography of Interpretation Evolution in LLM-Supported Programming Education](https://mlsanity.com/paper/2607.22463/) (2026)
- [Is Solving Better Than Evaluating GenAI Solutions?](https://mlsanity.com/paper/2607.27586/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards code reading, comprehension and evaluation of LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly for the first time in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.