mlsanityData to 14 Oct 2026

RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments

arXiv 2506.15253 ↗First posted 18 Jun 2025cs.CR cs.AIcited by 17

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The rapid deployment of Large language model (LLM) agents in critical domains like healthcare and finance necessitates robust security frameworks. To address the absence of standardized evaluation benchmarks for these agents in dynamic environments, we introduce RAS-Eval, a comprehensive security benchmark supporting both simulated and real-world tool execution. RAS-Eval comprises 80 test cases and 3,802 attack tasks mapped to 11 Common Weakness Enumeration (CWE) categories, with tools implemented in JSON, LangGraph, and Model Context Protocol (MCP) formats. We evaluate 6 state-of-the-art LLMs across diverse scenarios, revealing significant vulnerabilities: attacks reduced agent task completion rates (TCR) by 36.78% on average and achieved an 85.65% success rate in academic settings. Notably, scaling laws held for security capabilities, with larger models outperforming smaller counterparts. Our findings expose critical risks in real-world agent deployments and provide a foundational framework for future security research. Code and data are available at https://github.com/lanzer-tree/RAS-Eval.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 19 of its 23 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

19 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents31 May
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  4. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 15

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  7. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  8. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  9. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
  10. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  11. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  12. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 63
3 more
  1. 2025Enterprise-Grade Security for the Model Context Protocol (MCP): Frameworks and Mitigation Strategiescited by 42
  2. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 34
  3. 2025Cerebrum (AIOS SDK): A Platform for Agent Development, Deployment, Distribution, and Discoverycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  2. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  3. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7
  4. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 6
  5. 2026CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agentscited by 5
  6. 2025Evaluating LLM Generated Detection Rules in Cybersecuritycited by 4
  7. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 4
  8. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  9. 2026ADR: An Agentic Detection System for Enterprise Agentic AI Securitycited by 2
  10. 2026AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectoriescited by 2
  11. 2025EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Lawcited by 1
  12. 2026WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Taskscited by 1
5 more
  1. 2026Content-Aware Attack Detection in LLM Agent Tool-Call Traffic: An Empirical Study of Features, Architectures, and Evaluation Protocolscited by 1
  2. 2026When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agentscited by 1
  3. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilitiescited by 1
  4. 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluationcited by 0
  5. 2026Deny Without Disabling: Authorization-Paired Evaluation and Control for Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  2. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  3. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  4. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
  5. 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms
  6. 2025SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents
  7. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
  8. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems