# Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcare

arXiv: https://arxiv.org/abs/2606.28666 · HTML: https://mlsanity.com/paper/2606.28666/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 24 of 24)

- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [LLM Dataset Inference: Did you train on my dataset?](https://mlsanity.com/paper/2406.06443/) (2024)
- [Language Model Inversion](https://mlsanity.com/paper/2311.13647/) (2023)
- [TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems](https://mlsanity.com/paper/2506.04133/) (2025)
- [Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks](https://mlsanity.com/paper/2502.08586/) (2025)
- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Securing the Model Context Protocol (MCP): Risks, Controls, and Governance](https://mlsanity.com/paper/2511.20920/) (2025)
- [Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks](https://mlsanity.com/paper/2512.03262/) (2025)
- [Security Threats in Agentic AI System](https://mlsanity.com/paper/2410.14728/) (2024)
- [Evaluation of Inference Attack Models for Deep Learning on Medical Data](https://mlsanity.com/paper/2011.00177/) (2020)
- [Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks](https://mlsanity.com/paper/2509.22486/) (2025)
- [Operationalizing Data Minimization for Privacy-Preserving LLM Prompting](https://mlsanity.com/paper/2510.03662/) (2025)
- [Securing AI Agents Against Prompt Injection Attacks](https://mlsanity.com/paper/2511.15759/) (2025)
- [Memory poisoning and secure multi-agent systems](https://mlsanity.com/paper/2603.20357/) (2026)
- [MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs](https://mlsanity.com/paper/2602.06268/) (2026)
- [Analysing Environmental Efficiency in AI for X-Ray Diagnosis](https://mlsanity.com/paper/2511.07436/) (2025)
- [AI TIPS 2.0: A Comprehensive Framework for Operationalizing AI Governance](https://mlsanity.com/paper/2512.09114/) (2025)

## Abstract (arXiv)

Agent-based AI has enabled the automation of tasks by exposing application tools and resources to large language models (LLMs). However, to improve scope and accuracy, agents are often given access rights that exceed those of ordinary users, introducing significant security risks. AI is routinely integrated into applications with a disregard to security, risking data exposure and breaching regulations. This paper applies the AI Trust, Risk, and Security Management (TRiSM) framework to a medical report-generation application to demonstrate how an insecure agent workflow can be transformed into security-conscious agentic workflow. Both workflows were evaluated across five LLMs (Claude Haiku 4.5, GPT-4.1-nano, GPT-4.1-mini, GPT-5.4-mini, and Gemini 2.5 Flash) on two report types, totalling 800 generations and 500 attack scenarios including RAG poisoning, data-field injection, and client-side network injection. The TRiSM-guided agentic workflow reduced mean attack success rates from 31% to 10% for RAG poisoning and from 42% to 25% for data-field injection, while eliminating the network injection vector entirely through server-side prompt construction. Furthermore, report accuracy increased by 14 percentage points (72.5% to 86.5%) with the agentic workflow, demonstrating a secure design which provides more reliable outputs. This paper contributes to knowledge by demonstrating least-privilege, defence in depth agentic workflows improving security and accuracy, while also highlighting model choice is a necessary architectural consideration.