# Prompt Coverage Adequacy

arXiv: https://arxiv.org/abs/2607.02057 · HTML: https://mlsanity.com/paper/2607.02057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review](https://mlsanity.com/paper/2010.10596/) (2020)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024)
- [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015)
- [Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review](https://mlsanity.com/paper/2510.00328/) (2025)
- [Semantically Enhanced Software Traceability Using Deep Learning Techniques](https://mlsanity.com/paper/1804.02438/) (2018)
- [Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering](https://mlsanity.com/paper/2505.12025/) (2025)
- [Evaluation of large language models for assessing code maintainability](https://mlsanity.com/paper/2401.12714/) (2024)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [How well LLM-based test generation techniques perform with newer LLM versions?](https://mlsanity.com/paper/2601.09695/) (2026)

## Abstract (arXiv)

In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the focus from writing precise procedures to expressing intents and goals. This paradigm shift introduces new challenges, particularly in how testing should be guided when prompts, rather than code, become primary development artifacts. To address this challenge, we propose Prompt Coverage Adequacy, a novel coverage criterion designed to support the testing of code generated from task descriptions. Prompt Coverage Adequacy serves as an analog to traditional code coverage, but operates at the level of prompts used in LLM and agent-based programming. Specifically, it measures how well a given test suite satisfies the requirements expressed in a prompt by leveraging the attention mechanisms of LLMs. We evaluate a simple instantiation of this criterion, based on attention boosting, across two datasets and multiple LLMs. Our results demonstrate that Prompt Coverage is associated with fault-detection effectiveness and can uncover over 30+% more faults than traditional code coverage when used to guide test generation. These findings suggest that Prompt Coverage Adequacy can serve as a foundation for developing testing metrics better suited to the emerging paradigm of LLM-driven software development, addressing the limitations of classical coverage criteria in this new context.