# PromptSet: A Programmer's Prompting Dataset

arXiv: https://arxiv.org/abs/2402.16932 · HTML: https://mlsanity.com/paper/2402.16932/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2024.

## Built on by (most cited first, 9 of 9)

- [Benchmarking Human and Automated Prompting in the Segment Anything Model](https://mlsanity.com/paper/2410.22048/) (2024)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Prompts](https://mlsanity.com/paper/2501.12521/) (2025)
- [Secure Code Generation at Scale with Reflexion](https://mlsanity.com/paper/2511.03898/) (2025)
- [Reachability Across the NL/PL Boundary: A Taxonomy-Driven Dataflow Model for LLM-Integrated Applications](https://mlsanity.com/paper/2603.28345/) (2026)
- [Configuration Smells in AGENTS.md Files: Common Mistakes in Configuring Coding Agents](https://mlsanity.com/paper/2606.15828/) (2026)
- [IFHierBench: Hierarchical Instruction Following for Large Language Models](https://mlsanity.com/paper/2607.27912/) (2026)
- [Assertion-Aware Test Code Summarization with Large Language Models](https://mlsanity.com/paper/2511.06227/) (2025)
- [kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail](https://mlsanity.com/paper/2607.02072/) (2026)

## Abstract (arXiv)

The rise of capabilities expressed by large language models has been quickly followed by the integration of the same complex systems into application level logic. Algorithms, programs, systems, and companies are built around structured prompting to black box models where the majority of the design and implementation lies in capturing and quantifying the `agent mode'. The standard way to shape a closed language model is to prime it for a specific task with a tailored prompt, often initially handwritten by a human. The textual prompts co-evolve with the codebase, taking shape over the course of project life as artifacts which must be reviewed and maintained, just as the traditional code files might be. Unlike traditional code, we find that prompts do not receive effective static testing and linting to prevent runtime issues. In this work, we present a novel dataset called PromptSet, with more than 61,000 unique developer prompts used in open source Python programs. We perform analysis on this dataset and introduce the notion of a static linter for prompts. Released with this publication is a HuggingFace dataset and a Github repository to recreate collection and processing efforts, both under the name textttpisterlabs/promptset.