# Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment

arXiv: https://arxiv.org/abs/2603.07460 · HTML: https://mlsanity.com/paper/2603.07460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [Red-Teaming LLM Multi-Agent Systems via Communication Attacks](https://mlsanity.com/paper/2502.14847/) (2025)
- [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2402.00898/) (2024)
- [To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt](https://mlsanity.com/paper/2506.05739/) (2025)
- [Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal](https://mlsanity.com/paper/2403.13309/) (2024)
- [Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications](https://mlsanity.com/paper/2406.11007/) (2024)
- [Prompt Injection Attacks on Large Language Models in Oncology](https://mlsanity.com/paper/2407.18981/) (2024)
- [Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions](https://mlsanity.com/paper/2603.03637/) (2026)
- [A Survey of Automatic Generation of Attack Trees and Attack Graphs](https://mlsanity.com/paper/2302.14479/) (2023)
- [Security for Emerging Miniaturized Wireless Biomedical Devices: Threat Modeling with Application to Case Studies](https://mlsanity.com/paper/2105.05937/) (2021)
- [On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs](https://mlsanity.com/paper/2412.20087/) (2024)
- [A limited technical background is sufficient for attack-defense tree acceptability](https://mlsanity.com/paper/2502.11920/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly integrated into safety-critical workflows, yet existing security analyses remain fragmented and often isolate model behavior from the broader system context. This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS)-based exploitability scoring to support structured, comparable analysis. We demonstrate the framework through a healthcare case study, modeling multi-step attack paths targeting intervention in medical procedures, leakage of electronic health record (EHR) data, and disruption of service availability. Our analysis indicates that threats spanning (i) conventional cyber, (ii) adversarial ML, and (iii) conversational attacks that manipulate prompts or context often consolidate into a small number of dominant paths and shared system choke points, enabling targeted defenses to yield meaningful reductions in path exploitability. By systematically comparing defense portfolios, we align these risks with established vulnerability management practices and provide a domain-agnostic workflow applicable to other LLM-enabled critical systems.