mlsanityData to 14 Oct 2026

Evaluating LLM Generated Detection Rules in Cybersecurity

arXiv 2509.16749 ↗First posted 20 Sep 2025cs.CR cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

LLMs are increasingly pervasive in the security environment, with limited measures of their effectiveness, which limits trust and usefulness to security practitioners. Here, we present an open-source evaluation framework and benchmark metrics for evaluating LLM-generated cybersecurity rules. The benchmark employs a holdout set-based methodology to measure the effectiveness of LLM-generated security rules in comparison to a human-generated corpus of rules. It provides three key metrics inspired by the way experts evaluate security rules, offering a realistic, multifaceted evaluation of the effectiveness of an LLM-based security rule generator. This methodology is illustrated using rules from Sublime Security's detection team and those written by Sublime Security's Automated Detection Engineer (ADE), with a thorough analysis of ADE's skills presented in the results section.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 21 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

19 of 21 references.

References · 19

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  5. 2024AI Agents That Mattercited by 219
  6. 2024LLM Agents can Autonomously Hack Websitescited by 109
  7. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  8. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  9. 2025SWE-bench Goes Live!cited by 80
  10. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  11. 2024Constrained Decoding for Secure Code Generationcited by 37
  12. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
7 more
  1. 2025RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environmentscited by 17
  2. 2024When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?cited by 12
  3. 2025AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Modelscited by 7
  4. 2025A Preliminary Study on the Robustness of Code Generation by Large Language Modelscited by 5
  5. 2025DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environmentscited by 4
  6. 2025A Practical Guide for Evaluating LLMs and LLM-Reliant Systemscited by 3
  7. 2024Evaluating and Improving the Robustness of Security Attack Detectors Generated by LLMscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilitiescited by 4
  2. 2026SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classificationcited by 0
  3. 2026Operationalizing Cyber Threat Intelligence with GraphRAGcited by 0
  4. 2026Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesiscited by 0