mlsanityData to 7 Oct 2026

A Jailbroken GenAI Model Can Cause Substantial Harm: GenAI-powered Applications are Vulnerable to PromptWares

arXiv 2408.05061 ↗First posted 9 Aug 2024cs.CR cs.AIcited by 6

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

In this paper we argue that a jailbroken GenAI model can cause substantial harm to GenAI-powered applications and facilitate PromptWare, a new type of attack that flips the GenAI model's behavior from serving an application to attacking it. PromptWare exploits user inputs to jailbreak a GenAI model to force/perform malicious activity within the context of a GenAI-powered application. First, we introduce a naive implementation of PromptWare that behaves as malware that targets Plan & Execute architectures (a.k.a., ReAct, function calling). We show that attackers could force a desired execution flow by creating a user input that produces desired outputs given that the logic of the GenAI-powered application is known to attackers. We demonstrate the application of a DoS attack that triggers the execution of a GenAI-powered assistant to enter an infinite loop that wastes money and computational resources on redundant API calls to a GenAI engine, preventing the application from providing service to a user. Next, we introduce a more sophisticated implementation of PromptWare that we name Advanced PromptWare Threat (APwT) that targets GenAI-powered applications whose logic is unknown to attackers. We show that attackers could create user input that exploits the GenAI engine's advanced AI capabilities to launch a kill chain in inference time consisting of six steps intended to escalate privileges, analyze the application's context, identify valuable assets, reason possible malicious activities, decide on one of them, and execute it. We demonstrate the application of APwT against a GenAI-powered e-commerce chatbot and show that it can trigger the modification of SQL tables, potentially leading to unauthorized discounts on the items sold to the user.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 23 of its 47 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

23 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 20

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  4. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  5. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  6. 2023An LLM Compiler for Parallel Function Callingcited by 170
  7. 2024Stealing Part of a Production Language Modelcited by 152
  8. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  9. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  10. 2024LLM Agents can Autonomously Hack Websitescited by 108
  11. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  12. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
8 more
  1. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  2. 2023Effective Prompt Extraction from Language Modelscited by 83
  3. 2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 82
  4. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  5. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 58
  6. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  7. 2023Demystifying RCE Vulnerabilities in LLM-Integrated Appscited by 35
  8. 2024What Was Your Prompt? A Remote Keylogging Attack on AI Assistantscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  2. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 7
  3. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  4. 2024Generating a Low-code Complete Workflow via Task Decomposition and RAGcited by 3
  5. 2024The Best Defense is a Good Offense: Countering LLM-Powered Cyberattackscited by 2
  6. 2025Model Risk Management for Generative AI In Financial Institutionscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  2. 2025Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking
  3. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  4. 2026Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
  5. 2025Jailbreaking in the Haystack
  6. 2024EnJa: Ensemble Jailbreak on Large Language Models
  7. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  8. 2023Identifying and Mitigating the Security Risks of Generative AI