# Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworks

arXiv: https://arxiv.org/abs/2512.14860 · HTML: https://mlsanity.com/paper/2512.14860/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 12 of 12)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [Lessons from Defending Gemini Against Indirect Prompt Injections](https://mlsanity.com/paper/2505.14534/) (2025)
- [Security of AI Agents](https://mlsanity.com/paper/2406.08689/) (2024)
- [Seven Security Challenges in Cross-domain Multi-agent LLM Systems](https://mlsanity.com/paper/2505.23847/) (2025)
- [Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms](https://mlsanity.com/paper/2507.06323/) (2025)

## Built on by (most cited first, 4 of 4)

- [Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents](https://mlsanity.com/paper/2609.28940/) (2026)
- [From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation](https://mlsanity.com/paper/2609.22664/) (2026)
- [Fully Homomorphic Encryption on Llama 3 model for privacy preserving LLM inference](https://mlsanity.com/paper/2604.12168/) (2026)
- [Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges](https://mlsanity.com/paper/2609.23894/) (2026)

## Abstract (arXiv)

Agentic AI introduces security vulnerabilities that traditional LLM safeguards fail to address. Although recent work by Unit 42 at Palo Alto Networks demonstrated that ChatGPT-4o successfully executes attacks as an agent that it refuses in chat mode, there is no comparative analysis in multiple models and frameworks. We conducted the first systematic penetration testing and comparative evaluation of agentic AI systems, testing five prominent models (Claude 3.5 Sonnet, Gemini 2.5 Flash, GPT-4o, Grok 2, and Nova Pro) across two agentic AI frameworks (AutoGen and CrewAI) using a seven-agent architecture that mimics the functionality of a university information management system and 13 distinct attack scenarios that span prompt injection, Server Side Request Forgery (SSRF), SQL injection, and tool misuse. Our 130 total test cases reveal significant security disparities: AutoGen demonstrates a 52.3% refusal rate versus CrewAI's 30.8%, while model performance ranges from Nova Pro's 46.2% to Claude and Grok 2's 38.5%. Most critically, Grok 2 on CrewAI rejected only 2 of 13 attacks (15.4% refusal rate), and the overall refusal rate of 41.5% across all configurations indicates that more than half of malicious prompts succeeded despite enterprise-grade safety mechanisms. We identify six distinct defensive behavior patterns including a novel "hallucinated compliance" strategy where models fabricate outputs rather than executing or refusing attacks, and provide actionable recommendations for secure agent deployment. Complete attack prompts are also included in the Appendix to enable reproducibility.