mlsanityData to 7 Oct 2026

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

arXiv 2603.13517 ↗First posted 13 Mar 2026cs.CRcited by 2

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33% of the performance gap between smaller and larger models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 8 of its 20 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

8 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents28 Oct

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  5. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoningcited by 12
  6. 2025Evaluating LLM Generated Detection Rules in Cybersecuritycited by 4
  7. 2025CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classificationcited by 0
  2. 2026Operationalizing Cyber Threat Intelligence with GraphRAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
  2. 2026CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
  3. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
  4. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
  5. 2026CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking
  6. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence
  7. 2026SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
  8. 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents