mlsanityData to 7 Oct 2026

Detection of adversarial intent in Human-AI teams using LLMs

arXiv 2603.20976 ↗First posted 21 Mar 2026cs.LG cs.AI cs.HCcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in human-AI teams as support agents for complex tasks such as information retrieval, programming, and decision-making assistance. While these agents' autonomy and contextual knowledge enables them to be useful, it also exposes them to a broad range of attacks, including data poisoning, prompt injection, and even prompt engineering. Through these attack vectors, malicious actors can manipulate an LLM agent to provide harmful information, potentially manipulating human agents to make harmful decisions. While prior work has focused on LLMs as attack targets or adversarial actors, this paper studies their potential role as defensive supervisors within mixed human-AI teams. Using a dataset consisting of multi-party conversations and decisions for a real human-AI team over a 25 round horizon, we formulate the problem of malicious behavior detection from interaction traces. We find that LLMs are capable of identifying malicious behavior in real-time, and without task-specific information, indicating the potential for task-agnostic defense. Moreover, we find that the malicious behavior of interest is not easily identified using simple heuristics, further suggesting the introduction of LLM defenders could render human teams more robust to certain classes of attack.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 13 of its 20 references that are in the map. First posted on arXiv 21 Mar 2026.

Builds on

13 of 20 references.

References · 13

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2020A review on outlier/anomaly detection in time series datacited by 187
  5. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  6. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  7. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  8. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  9. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  10. 2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 16
  11. 2021Deceptive Reinforcement Learning for Privacy-Preserving Planningcited by 14
  12. 2022Sparse Adversarial Attack in Multi-agent Reinforcement Learningcited by 8
1 more
  1. 2025Learning to Lie: Reinforcement Learning Attacks Damage Human-AI Teams and Teams of LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation
  2. 2023LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing
  3. 2025A Survey of Attacks on Large Language Models
  4. 2025Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilities
  5. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  6. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents
  7. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  8. 2026A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents