mlsanityData to 7 Oct 2026

A Taxonomy of Prompt Defects in LLM Systems

arXiv 2509.14404 ↗First posted 17 Sep 2025cs.SE cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have become key components of modern software, with prompts acting as their de-facto programming interface. However, prompt design remains largely empirical and small mistakes can cascade into unreliable, insecure, or inefficient behavior. This paper presents the first systematic survey and taxonomy of prompt defects, recurring ways that prompts fail to elicit their intended behavior from LLMs. We organize defects along six dimensions: (1) Specification and Intent, (2) Input and Content, (3) Structure and Formatting, (4) Context and Memory, (5) Performance and Efficiency, and (6) Maintainability and Engineering. Each dimension is refined into fine-grained subtypes, illustrated with concrete examples and root cause analysis. Grounded in software engineering principles, we show how these defects surface in real development workflows and examine their downstream effects. For every subtype, we distill mitigation strategies that span emerging prompt engineering patterns, automated guardrails, testing harnesses, and evaluation frameworks. We then summarize these strategies in a master taxonomy that links defect, impact, and remedy. We conclude with open research challenges and a call for rigorous engineering-oriented methodologies to ensure that LLM-driven systems are dependable by design.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

31 of 45 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  10. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
19 more
  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  3. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  4. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  5. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
  6. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  7. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  8. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  9. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
  10. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  11. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  12. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  13. 2025Enhancing repository-level software repair via repository-aware knowledge graphscited by 26
  14. 2025The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMscited by 19
  15. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  16. 2024MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-tuningcited by 13
  17. 2024Are LLMs Correctly Integrated into Software Systems?cited by 13
  18. 2024Are Longer Prompts Always Better? Prompt Selection in Large Language Models for Recommendation Systemscited by 6
  19. 2023Prompt Sapper: LLM-Empowered Software Engineering Infrastructure for AI-Native Servicescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.