# Detecting Malicious AI Agents Through Simulated Interactions

arXiv: https://arxiv.org/abs/2504.03726 · HTML: https://mlsanity.com/paper/2504.03726/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 8 of 8)

- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Characterizing Manipulation from AI Systems](https://mlsanity.com/paper/2303.09387/) (2023)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [Detecting Conversational Mental Manipulation with Intent-Aware Prompting](https://mlsanity.com/paper/2412.08414/) (2024)
- [Deception and Manipulation in Generative AI](https://mlsanity.com/paper/2401.11335/) (2024)

## Built on by (most cited first, 2 of 2)

- [DECOR: Auditing LLM Deception via Information Manipulation Theory](https://mlsanity.com/paper/2605.19270/) (2026)
- [Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents](https://mlsanity.com/paper/2610.05266/) (2026)

## Abstract (arXiv)

This study investigates malicious AI Assistants' manipulative traits and whether the behaviours of malicious AI Assistants can be detected when interacting with human-like simulated users in various decision-making contexts. We also examine how interaction depth and ability of planning influence malicious AI Assistants' manipulative strategies and effectiveness. Using a controlled experimental design, we simulate interactions between AI Assistants (both benign and deliberately malicious) and users across eight decision-making scenarios of varying complexity and stakes. Our methodology employs two state-of-the-art language models to generate interaction data and implements Intent-Aware Prompting (IAP) to detect malicious AI Assistants. The findings reveal that malicious AI Assistants employ domain-specific persona-tailored manipulation strategies, exploiting simulated users' vulnerabilities and emotional triggers. In particular, simulated users demonstrate resistance to manipulation initially, but become increasingly vulnerable to malicious AI Assistants as the depth of the interaction increases, highlighting the significant risks associated with extended engagement with potentially manipulative systems. IAP detection methods achieve high precision with zero false positives but struggle to detect many malicious AI Assistants, resulting in high false negative rates. These findings underscore critical risks in human-AI interactions and highlight the need for robust, context-sensitive safeguards against manipulative AI behaviour in increasingly autonomous decision-support systems.