# Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories

arXiv: https://arxiv.org/abs/2412.17298 · HTML: https://mlsanity.com/paper/2412.17298/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2024.

## Built on by (most cited first, 18 of 18)

- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills](https://mlsanity.com/paper/2607.01456/) (2026)
- [Understanding Prompt Management in GitHub Repositories: A Call for Best Practices](https://mlsanity.com/paper/2509.12421/) (2025)
- [Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair](https://mlsanity.com/paper/2510.03217/) (2025)
- [Exploring Language Patterns of Prompts in Text-to-Image Generation and Their Impact on Visual Diversity](https://mlsanity.com/paper/2504.14125/) (2025)
- [Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding](https://mlsanity.com/paper/2608.11095/) (2026)
- [The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SE](https://mlsanity.com/paper/2604.15468/) (2026)
- [Understanding Prompt Programming Tasks and Questions](https://mlsanity.com/paper/2507.17264/) (2025)
- [Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering](https://mlsanity.com/paper/2509.17096/) (2025)
- [Secure Code Generation at Scale with Reflexion](https://mlsanity.com/paper/2511.03898/) (2025)
- [Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance](https://mlsanity.com/paper/2604.17650/) (2026)
- [An Empirical Study of Downstream Adaptation for Agent Skills](https://mlsanity.com/paper/2607.03238/) (2026)
- [Assertion-Aware Test Code Summarization with Large Language Models](https://mlsanity.com/paper/2511.06227/) (2025)
- [Adoption of Generative Artificial Intelligence in the German Software Engineering Industry: An Empirical Study](https://mlsanity.com/paper/2601.16700/) (2026)
- [Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code](https://mlsanity.com/paper/2608.12905/) (2026)
- [Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance](https://mlsanity.com/paper/2609.05677/) (2026)
- [Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows](https://mlsanity.com/paper/2609.27263/) (2026)
- [RLPE: Teaching LLMs to Prompt with Reinforcement Learning](https://mlsanity.com/paper/2610.19062/) (2026)

## Abstract (arXiv)

The adoption of Large Language Models (LLMs) is reshaping software development as developers integrate these LLMs into their applications. In such applications, prompts serve as the primary means of interacting with LLMs. Despite the widespread use of LLM-integrated applications, there is limited understanding of how developers manage and evolve prompts. This study presents the first empirical analysis of prompt evolution in LLM-integrated software development. We analyzed 1,262 prompt changes across 243 GitHub repositories to investigate the patterns and frequencies of prompt changes, their relationship with code changes, documentation practices, and their impact on system behavior. Our findings show that developers primarily evolve prompts through additions and modifications, with most changes occurring during feature development. We identified key challenges in prompt engineering: only 21.9% of prompt changes are documented in commit messages, changes can introduce logical inconsistencies, and misalignment often occurs between prompt changes and LLM responses. These insights emphasize the need for specialized testing frameworks, automated validation tools, and improved documentation practices to enhance the reliability of LLM-integrated applications.