mlsanityData to 7 Oct 2026

DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments

arXiv 2506.00739 ↗First posted 31 May 2025cs.CLcited by 4

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language model (LLM) agents have shown impressive capabilities in human language comprehension and reasoning, yet their potential in cybersecurity remains underexplored. We introduce DefenderBench, a practical, open-source toolkit for evaluating language agents across offense, defense, and cybersecurity knowledge-based tasks. DefenderBench includes environments for network intrusion, malicious content detection, code vulnerability analysis, and cybersecurity knowledge assessment. It is intentionally designed to be affordable and easily accessible for researchers while providing fair and rigorous assessment. We benchmark several state-of-the-art (SoTA) and popular LLMs, including both open- and closed-weight models, using a standardized agentic framework. Our results show that Claude-3.7-sonnet performs best with a DefenderBench score of 81.65, followed by Claude-3.7-sonnet-think with 78.40, while the best open-weight model, Llama 3.3 70B, is not far behind with a DefenderBench score of 71.81. DefenderBench's modular design allows seamless integration of custom LLMs and tasks, promoting reproducibility and fair comparisons. An anonymized version of DefenderBench is available at https://github.com/microsoft/DefenderBench.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 40 of its 53 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

40 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments17 Sep
  2. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilities11 Apr
  3. 2024LLM Agents can Autonomously Hack Websites6 Feb

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
25 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  10. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  11. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  12. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  13. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  14. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  15. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  16. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 94
  17. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 89
  18. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  19. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  20. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  21. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  22. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  23. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  24. 2024UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debuggingcited by 35
  25. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Evaluating LLM Generated Detection Rules in Cybersecuritycited by 4
  2. 2025The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelinescited by 2
  3. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  4. 2026KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding
  2. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence
  3. 2025From Texts to Shields: Convergence of Large Language Models and Cybersecurity
  4. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  5. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
  6. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  7. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations
  8. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense