# Understanding Prompt Programming Tasks and Questions

arXiv: https://arxiv.org/abs/2507.17264 · HTML: https://mlsanity.com/paper/2507.17264/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Prompt Engineering a Prompt Engineer](https://mlsanity.com/paper/2311.05661/) (2023)
- ["We Need Structured Output": Towards User-centered Constraints on Large Language Model Output](https://mlsanity.com/paper/2404.07362/) (2024)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps](https://mlsanity.com/paper/2504.02052/) (2025)
- [Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts](https://mlsanity.com/paper/2409.12447/) (2024)
- [Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories](https://mlsanity.com/paper/2412.17298/) (2024)
- [Building Your Own Product Copilot: Challenges, Opportunities, and Needs](https://mlsanity.com/paper/2312.14231/) (2023)
- [CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming](https://mlsanity.com/paper/2310.09235/) (2023)
- [WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI](https://mlsanity.com/paper/2308.13355/) (2023)
- [Understanding Skills for OSS Communities on GitHub](https://mlsanity.com/paper/2209.02222/) (2022)
- [PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation](https://mlsanity.com/paper/2410.16547/) (2024)
- [CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering](https://mlsanity.com/paper/2411.06099/) (2024)
- [Leaving My Fingerprints: Motivations and Challenges of Contributing to OSS for Social Good](https://mlsanity.com/paper/2104.12891/) (2021)
- [Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets](https://mlsanity.com/paper/2503.21394/) (2025)

## Built on by (most cited first, 1 of 1)

- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)

## Abstract (arXiv)

Prompting foundation models (FMs) like large language models (LLMs) have enabled new AI-powered software features (e.g., text summarization) that previously were only possible by fine-tuning FMs. Now, developers are embedding prompts in software, known as prompt programs. The process of prompt programming requires the developer to make many changes to their prompt. Yet, the questions developers ask to update their prompt is unknown, despite the answers to these questions affecting how developers plan their changes. With the growing number of research and commercial prompt programming tools, it is unclear whether prompt programmers' needs are being adequately addressed. We address these challenges by developing a taxonomy of 25 tasks prompt programmers do and 51 questions they ask, measuring the importance of each task and question. We interview 16 prompt programmers, observe 8 developers make prompt changes, and survey 50 developers. We then compare the taxonomy with 48 research and commercial tools. We find that prompt programming is not well-supported: all tasks are done manually, and 16 of the 51 questions -- including a majority of the most important ones -- remain unanswered. Based on this, we outline important opportunities for prompt programming tools.