# Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

arXiv: https://arxiv.org/abs/2410.20911 · HTML: https://mlsanity.com/paper/2410.20911/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 14 of 14)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks](https://mlsanity.com/paper/2403.03792/) (2024)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?](https://mlsanity.com/paper/2310.09810/) (2023)
- [An Empirical Evaluation of LLMs for Solving Offensive Security Challenges](https://mlsanity.com/paper/2402.11814/) (2024)
- [LLMs for Cyber Security: New Opportunities](https://mlsanity.com/paper/2404.11338/) (2024)
- [Hacking, The Lazy Way: LLM Augmented Pentesting](https://mlsanity.com/paper/2409.09493/) (2024)

## Built on by (most cited first, 15 of 15)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis](https://mlsanity.com/paper/2602.10453/) (2026)
- [From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem](https://mlsanity.com/paper/2506.15170/) (2025)
- [CyberRAG: An Agentic RAG cyber attack classification and reporting tool](https://mlsanity.com/paper/2507.02424/) (2025)
- [ARACNE: An LLM-Based Autonomous Shell Pentesting Agent](https://mlsanity.com/paper/2502.18528/) (2025)
- [Agentic AI and the Cyber Arms Race](https://mlsanity.com/paper/2503.04760/) (2025)
- [Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems](https://mlsanity.com/paper/2606.20470/) (2026)
- [AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents](https://mlsanity.com/paper/2607.26998/) (2026)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [Challenges and Future Directions in Agentic Reverse Engineering Systems](https://mlsanity.com/paper/2604.14317/) (2026)
- [Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility](https://mlsanity.com/paper/2511.13725/) (2025)
- [Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection](https://mlsanity.com/paper/2604.18248/) (2026)
- [Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents](https://mlsanity.com/paper/2606.24402/) (2026)
- [Red-Teaming the Agentic Red-Team](https://mlsanity.com/paper/2606.24496/) (2026)
- [Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery](https://mlsanity.com/paper/2609.35909/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being harnessed to automate cyberattacks, making sophisticated exploits more accessible and scalable. In response, we propose a new defense strategy tailored to counter LLM-driven cyberattacks. We introduce Mantis, a defensive framework that exploits LLMs' susceptibility to adversarial inputs to undermine malicious operations. Upon detecting an automated cyberattack, Mantis plants carefully crafted inputs into system responses, leading the attacker's LLM to disrupt their own operations (passive defense) or even compromise the attacker's machine (active defense). By deploying purposefully vulnerable decoy services to attract the attacker and using dynamic prompt injections for the attacker's LLM, Mantis can autonomously hack back the attacker. In our experiments, Mantis consistently achieved over 95% effectiveness against automated LLM-driven attacks. To foster further research and collaboration, Mantis is available as an open-source tool: https://github.com/pasquini-dario/project_mantis