# PromptPex: Automatic Test Generation for Language Model Prompts arXiv: https://arxiv.org/abs/2503.05070 ยท HTML: https://mlsanity.com/paper/2503.05070/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 95 references that are in the map. - First posted on arXiv 7 Mar 2025. ## Builds on (most cited first, 26 of 26) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022) - [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023) - [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023) - [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023) - [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024) - [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023) - [Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement](https://mlsanity.com/paper/2402.11436/) (2024) - [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023) - [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023) - [FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2309.05274/) (2023) - [SPML: A DSL for Defending Language Models Against Prompt Attacks](https://mlsanity.com/paper/2402.11755/) (2024) - [On the Evaluation of Large Language Models in Unit Test Generation](https://mlsanity.com/paper/2406.18181/) (2024) - [ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation](https://mlsanity.com/paper/2307.00588/) (2023) - [Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization](https://mlsanity.com/paper/2404.02319/) (2024) - [(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs](https://mlsanity.com/paper/2311.11123/) (2023) - [MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization](https://mlsanity.com/paper/2407.04118/) (2024) - [ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP](https://mlsanity.com/paper/2308.02122/) (2023) - [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024) - [GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agency](https://mlsanity.com/paper/2402.08855/) (2024) - [PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs](https://mlsanity.com/paper/2409.14729/) (2024) - [Specifications: The missing link to making the development of LLM systems an engineering discipline](https://mlsanity.com/paper/2412.05299/) (2024) - [Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools](https://mlsanity.com/paper/2312.10622/) (2023) - [Test Oracle Automation in the era of LLMs](https://mlsanity.com/paper/2405.12766/) (2024) - [RETAIN: Interactive Tool for Regression Testing Guided LLM Migration](https://mlsanity.com/paper/2409.03928/) (2024) ## Built on by (most cited first, 9 of 9) - [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025) - [Willful Disobedience: Automatically Detecting Failures in Agentic Traces](https://mlsanity.com/paper/2603.23806/) (2026) - [Sharing State Between Prompts and Programs](https://mlsanity.com/paper/2512.14805/) (2025) - [Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs](https://mlsanity.com/paper/2509.17314/) (2025) - [From Benchmark Data To Applicable Program Repair: An Experience Report](https://mlsanity.com/paper/2508.16071/) (2025) - [Prompt Less, Smile More: MTP with Semantic Engineering in Lieu of Prompt Engineering](https://mlsanity.com/paper/2511.19427/) (2025) - [Imprompt: A Language Framework for Prompt Programming](https://mlsanity.com/paper/2607.22683/) (2026) - [Agents as Software: A Programming Languages Agenda for Agent Reliability](https://mlsanity.com/paper/2609.32198/) (2026) - [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026) ## Abstract (arXiv) Large language models (LLMs) are being used in many applications and prompts for these models are integrated into software applications as code-like artifacts. These prompts behave much like traditional software in that they take inputs, generate outputs, and perform some specific function. However, prompts differ from traditional code in many ways and require new approaches to ensure that they are robust. For example, unlike traditional software the output of a prompt depends on the AI model that interprets it. Also, while natural language prompts are easy to modify, the impact of updates is harder to predict. New approaches to testing, debugging, and modifying prompts with respect to the model running them are required. To address some of these issues, we developed PromptPex, an LLM-based tool to automatically generate and evaluate unit tests for a given prompt. PromptPex extracts input and output specifications from a prompt and uses them to generate diverse, targeted, and valid unit tests. These tests are instrumental in identifying regressions when a prompt is changed and also serve as a tool to understand how prompts are interpreted by different models. We use PromptPex to generate tests for eight benchmark prompts and evaluate the quality of the generated tests by seeing if they can cause each of four diverse models to produce invalid output. PromptPex consistently creates tests that result in more invalid model outputs than a carefully constructed baseline LLM-based test generator. Furthermore, by extracting concrete specifications from the input prompt, PromptPex allows prompt writers to clearly understand and test specific aspects of their prompts. The source code of PromptPex is available at https://github.com/microsoft/promptpex.