mlsanityData to 13 Oct 2026

AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

arXiv 2602.16901 ↗First posted 18 Feb 2026cs.AIcited by 36

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

LLM agents are increasingly deployed in long-horizon, complex environments to solve challenging problems, but this expansion exposes them to long-horizon attacks that exploit multi-turn user-agent-environment interactions to achieve objectives infeasible in single-turn settings. To measure agent vulnerabilities to such risks, we present AgentLAB, the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks. Currently, AgentLAB supports five novel attack types including intent hijacking, tool chaining, task injection, objective drifting, and memory poisoning, spanning 28 realistic agentic environments, and 644 security test cases. Leveraging AgentLAB, we evaluate representative LLM agents and find that they remain highly susceptible to long-horizon attacks; moreover, defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats. We anticipate that AgentLAB will serve as a valuable benchmark for tracking progress on securing LLM agents in practical settings. The benchmark is publicly available at https://tanqiujiang.github.io/AgentLAB_main.

Key facts. Cited by 36 papers in the map (data through 13 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 32 of its 48 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

32 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection6 Oct
  2. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents26 Sep
  3. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents17 Jun
  4. 2025Prompt Injection Attack to Tool Selection in LLM Agents28 Apr
  5. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks22 Apr
  6. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  7. 2025Memory Injection Attacks on LLM Agents via Query-Only Interaction5 Mar
  8. 2025UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning28 Feb
  9. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  10. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents12 Nov
  11. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  12. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 20

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,117
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,219
  6. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,032
  7. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 760
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 384
  9. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 169
  10. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  11. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 123
  12. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 118
8 more
  1. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
  2. 2024Mitigating the Risk of Health Inequity Exacerbated by Large Language Modelscited by 21
  3. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  4. 2025CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Modelcited by 13
  5. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcarecited by 10
  6. 2025Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Modelscited by 10
  7. 2025DLRREC: Denoising Latent Representations via Multi-Modal Knowledge Fusion in Deep Recommender Systemscited by 5
  8. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 36.

36 papers.

Sort:

By area:

  1. 2026Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystemscited by 11
  2. 2026HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmarkcited by 9
  3. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulationcited by 9
  4. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  5. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environmentscited by 4
  6. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 4
  7. 2026Hijacking Agent Memory: Stealthy Trojan Attacks Through Conversational Interactioncited by 4
  8. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysiscited by 4
  9. 2026Cross-Layer Semantic Flow Reconstruction for Attack Detection in Agentic Systemscited by 3
  10. 2026CASPIAN: Online Detection and Attribution of Cascade Attacks in LLM Multi-Agent Systems via Cross-Channel Causal Monitoringcited by 3
  11. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  12. 2026Can LLMs Perceive Time? An Empirical Investigationcited by 2
24 more
  1. 2026When Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agentscited by 2
  2. 2026OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agentscited by 2
  3. 2026ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agentscited by 2
  4. 2026ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injectioncited by 2
  5. 2026CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinementcited by 2
  6. 2026TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safetycited by 1
  7. 2026JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safetycited by 1
  8. 2026ContainmentBench: Trace-Based Evaluation of Post-Exposure Containment in Tool-Using LLM Agentscited by 1
  9. 2026HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnessescited by 1
  10. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systemscited by 0
  11. 2026OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolutioncited by 0
  12. 2026Breadcrumbing Search Agentscited by 0
  13. 2026Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agentscited by 0
  14. 2026ECLIPSE: Self-Evolving Stealthy Prompt Injection Attack against Long-Horizon Agentic Systemscited by 0
  15. 2026BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agentscited by 0
  16. 2026Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systemscited by 0
  17. 2026Measuring and Exploiting Implicit Trust in LLM Tool-Calling Pipelinescited by 0
  18. 2026Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challengescited by 0
  19. 2026Persistent Billable State: Denial-of-Wallet Attacks and Defenses in Tool-Calling LLM Agentscited by 0
  20. 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agentscited by 0
  21. 2026Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?cited by 0
  22. 2026The Geometry of Harmfulness in Multi-Turn Attackscited by 0
  23. 2026Runtime Authorization of Self-Generated Subgoals in Long-Horizon Tool-Using AI Agentscited by 0
  24. 2026Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
  2. 2026Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
  3. 2026DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
  4. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents
  5. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  6. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memory
  7. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
  8. 2025A Survey on Agentic Security: Applications, Threats and Defenses