# Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

arXiv: https://arxiv.org/abs/2409.12447 · HTML: https://mlsanity.com/paper/2409.12447/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 83 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset](https://mlsanity.com/paper/2306.06826/) (2023)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges](https://mlsanity.com/paper/2012.07919/) (2020)
- [(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs](https://mlsanity.com/paper/2311.11123/) (2023)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models](https://mlsanity.com/paper/2304.01964/) (2023)
- [Building Your Own Product Copilot: Challenges, Opportunities, and Needs](https://mlsanity.com/paper/2312.14231/) (2023)
- [Understanding Skills for OSS Communities on GitHub](https://mlsanity.com/paper/2209.02222/) (2022)
- [Wordflow: Social Prompt Engineering for Large Language Models](https://mlsanity.com/paper/2401.14447/) (2024)
- [Development in times of hype: How freelancers explore Generative AI?](https://mlsanity.com/paper/2401.05790/) (2024)

## Built on by (most cited first, 19 of 19)

- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [Vibe Coding as a Reconfiguration of Intent Mediation in Software Development: Definition, Implications, and Research Agenda](https://mlsanity.com/paper/2507.21928/) (2025)
- [Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products](https://mlsanity.com/paper/2410.12071/) (2024)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)
- [Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective](https://mlsanity.com/paper/2511.06428/) (2025)
- [Sharing State Between Prompts and Programs](https://mlsanity.com/paper/2512.14805/) (2025)
- [Agentic Pipelines in Embedded Software Engineering: Emerging Practices and Challenges](https://mlsanity.com/paper/2601.10220/) (2026)
- [Understanding Prompt Programming Tasks and Questions](https://mlsanity.com/paper/2507.17264/) (2025)
- [Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering](https://mlsanity.com/paper/2509.17096/) (2025)
- [A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AI](https://mlsanity.com/paper/2510.26275/) (2025)
- [Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks](https://mlsanity.com/paper/2508.04125/) (2025)
- [The Limits of Goal-Setting Theory in LLM-Driven Assessment](https://mlsanity.com/paper/2510.06997/) (2025)
- [Semantic Content Determines Algorithmic Performance](https://mlsanity.com/paper/2601.21618/) (2026)
- ["I Just Need GPT to Refine My Prompts": Rethinking Onboarding and Help-Seeking with Generative 3D Modeling Tools](https://mlsanity.com/paper/2603.29118/) (2026)
- [Graphical-Probabilistic Modeling of Generative Flows in LLM-Native Software Systems](https://mlsanity.com/paper/2606.15943/) (2026)
- [Ceci n'est pas une pipe: AI systems as semantic abstractions](https://mlsanity.com/paper/2607.09489/) (2026)
- [Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution](https://mlsanity.com/paper/2608.21341/) (2026)

## Abstract (arXiv)

Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming". We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)'s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.