# GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

arXiv: https://arxiv.org/abs/2606.05844 · HTML: https://mlsanity.com/paper/2606.05844/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 157 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 24 of 24)

- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [LogGPT: Exploring ChatGPT for Log-Based Anomaly Detection](https://mlsanity.com/paper/2309.01189/) (2023)
- [Generating Fake Cyber Threat Intelligence Using Transformer-Based Models](https://mlsanity.com/paper/2102.04351/) (2021)
- [Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTI](https://mlsanity.com/paper/2211.01753/) (2022)
- [HuntGPT: Integrating Machine Learning-Based Anomaly Detection and Explainable AI with Large Language Models (LLMs)](https://mlsanity.com/paper/2309.16021/) (2023)
- [FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2304.14746/) (2023)
- [LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledge](https://mlsanity.com/paper/2401.10036/) (2024)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [CAN-BERT do it? Controller Area Network Intrusion Detection System based on BERT Language Model](https://mlsanity.com/paper/2210.09439/) (2022)
- [5G-NIDD: A Comprehensive Network Intrusion Detection Dataset Generated over 5G Wireless Network](https://mlsanity.com/paper/2212.01298/) (2022)
- [AGIR: Automating Cyber Threat Intelligence Reporting with Natural Language Generation](https://mlsanity.com/paper/2310.02655/) (2023)
- [SVGen: Interpretable Vector Graphics Generation with Large Language Models](https://mlsanity.com/paper/2508.09168/) (2025)
- [MITRE ATT&CK: State of the Art and Way Forward](https://mlsanity.com/paper/2308.14016/) (2023)
- [Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Study](https://mlsanity.com/paper/2208.12144/) (2022)
- [A Survey on Malware Detection with Graph Representation Learning](https://mlsanity.com/paper/2303.16004/) (2023)
- [On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions](https://mlsanity.com/paper/2306.14062/) (2023)
- [PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention network](https://mlsanity.com/paper/2311.12372/) (2023)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [LogPr'ecis: Unleashing Language Models for Automated Malicious Log Analysis](https://mlsanity.com/paper/2307.08309/) (2023)
- [Unknown Face Presentation Attack Detection via Localised Learning of Multiple Kernels](https://mlsanity.com/paper/2204.10675/) (2022)
- [Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack](https://mlsanity.com/paper/2503.15551/) (2025)
- [GPT-2C: A GPT-2 parser for Cowrie honeypot logs](https://mlsanity.com/paper/2109.06595/) (2021)
- [A Collaborative Intrusion Detection System Using Snort IDS Nodes](https://mlsanity.com/paper/2504.16550/) (2025)
- [GRIDAI: Generating and Repairing Intrusion Detection Rules via Collaboration among Multiple LLM-based Agents](https://mlsanity.com/paper/2510.13257/) (2025)

## Abstract (arXiv)

Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, signature-driven rules limit adaptability to emerging and zero-day threats. Additionally, existing public datasets (e.g., CICIDS2017, UNSW-NB15) focus on traffic classification and provide little structured information to support automatic rule synthesis or prevention logic. To address this gap, we propose Generative Thread Intelligence (GenTI) footnoteGenTI refers to the proposed framework, and GTI refers to the dataset. an LLM-driven benchmark for automatic generation of IDPS rules targeting unseen attacks. The dataset (GTI) aggregates over 150k detection and prevention rules from Snort, Suricata, Emerging Threats, as well as 50k YARA, each annotated with protocol behavior, payload signatures, contextual relationships, mappings to Cyber Threat Intelligence (CTI), along with actionable response types (alert, drop, reject). Moreover, on top of this corpus we design an LLM-based pipeline that transforms analyst prompts and representative payloads into deployable rules via structured prompt engineering, Chain-of-Thought (CoT) reasoning, as well as a Chain-of-Verification (CoVe) loop for syntactic, semantic, and security validation. The generated rules are executed in real time on (Snort/Suricata) and evaluated by syntax accuracy, semantic similarity, CTI coverage, security effectiveness as well as unseen attacks detection. Furthermore, our GenTI instantiation achieves a composite rule-quality score of 89.4%, with 94.8% CTI coverage, improving unseen attacks detection from 45% to 87.4% and reducing the false-positive rate from 8.5% to 2.3%. Overall, GenTI establishes the first large-scale benchmark that tightly couples rule-level CTI with LLM-based automation, enabling adaptive, self-evolving IDPS.