A Taxonomy of Prompt Defects in LLM Systems
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) have become key components of modern software, with prompts acting as their de-facto programming interface. However, prompt design remains largely empirical and small mistakes can cascade into unreliable, insecure, or inefficient behavior. This paper presents the first systematic survey and taxonomy of prompt defects, recurring ways that prompts fail to elicit their intended behavior from LLMs. We organize defects along six dimensions: (1) Specification and Intent, (2) Input and Content, (3) Structure and Formatting, (4) Context and Memory, (5) Performance and Efficiency, and (6) Maintainability and Engineering. Each dimension is refined into fine-grained subtypes, illustrated with concrete examples and root cause analysis. Grounded in software engineering principles, we show how these defects surface in real development workflows and examine their downstream effects. For every subtype, we distill mitigation strategies that span emerging prompt engineering patterns, automated guardrails, testing harnesses, and evaluation frameworks. We then summarize these strategies in a master taxonomy that links defect, impact, and remedy. We conclude with open research challenges and a call for rigorous engineering-oriented methodologies to ensure that LLM-driven systems are dependable by design.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 17 Sep 2025.
Builds on
31 of 45 references.
References · 31
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
19 more
- 2024Knowledge Conflicts for LLMs: A Surveycited by 274
- 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
- 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
- 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
- 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
- 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
- 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
- 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
- 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
- 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
- 2025Enhancing repository-level software repair via repository-aware knowledge graphscited by 26
- 2025The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMscited by 19
- 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
- 2024MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-tuningcited by 13
- 2024Are LLMs Correctly Integrated into Software Systems?cited by 13
- 2024Are Longer Prompts Always Better? Prompt Selection in Large Language Models for Recommendation Systemscited by 6
- 2023Prompt Sapper: LLM-Empowered Software Engineering Infrastructure for AI-Native Servicescited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 6.
7 papers.
- 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
- 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 13
- 2026Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradientscited by 5
- 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomycited by 5
- 2025Specification and Detection of LLM Code Smellscited by 0
- 2026Semantic Feature Analysis: Improving Agents Without Searching Over Rolloutscited by 0
- 2026Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Codecited by 0
- 2026Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code2 Jul 2026
- 2026Semantic Feature Analysis: Improving Agents Without Searching Over Rollouts12 Apr 2026
- 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments24 Mar 2026
- 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy5 Mar 2026
- 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis11 Feb 2026
- 2026Modular Prompt Optimization: Optimizing Structured Prompts with Section-Local Textual Gradients7 Jan 2026
- 2025Specification and Detection of LLM Code Smells19 Dec 2025