# Goal-Driven Risk Assessment for LLM-Powered Systems: A Healthcare Case Study

arXiv: https://arxiv.org/abs/2603.03633 · HTML: https://mlsanity.com/paper/2603.03633/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Red-Teaming LLM Multi-Agent Systems via Communication Attacks](https://mlsanity.com/paper/2502.14847/) (2025)
- [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2402.00898/) (2024)
- [Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal](https://mlsanity.com/paper/2403.13309/) (2024)
- [Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications](https://mlsanity.com/paper/2406.11007/) (2024)
- [Prompt Injection Attacks on Large Language Models in Oncology](https://mlsanity.com/paper/2407.18981/) (2024)
- [Security for Emerging Miniaturized Wireless Biomedical Devices: Threat Modeling with Application to Case Studies](https://mlsanity.com/paper/2105.05937/) (2021)
- [A limited technical background is sufficient for attack-defense tree acceptability](https://mlsanity.com/paper/2502.11920/) (2025)

## Abstract (arXiv)

While incorporating LLMs into systems offers significant benefits in critical application areas such as healthcare, new security challenges emerge due to the potential cyber kill chain cycles that combine adversarial model, prompt injection and conventional cyber attacks. Threat modeling methods enable the system designers to identify potential cyber threats and the relevant mitigations during the early stages of development. Although the cyber security community has extensive experience in applying these methods to software-based systems, the elicited threats are usually abstract and vague, limiting their effectiveness for conducting proper likelihood and impact assessments for risk prioritization, especially in complex systems with novel attacks surfaces, such as those involving LLMs. In this study, we propose a structured, goal driven risk assessment approach that contextualizes the threats with detailed attack vectors, preconditions, and attack paths through the use of attack trees. We demonstrate the proposed approach on a case study with an LLM agent-based healthcare system. This study harmonizes the state-of-the-art attacks to LLMs with conventional ones and presents possible attack paths applicable to similar systems. By providing a structured risk assessment, this study makes a significant contribution to the literature and advances the secure-by-design practices in LLM-based systems.