# On Automating Security Policies with Contemporary LLMs

arXiv: https://arxiv.org/abs/2506.04838 · HTML: https://mlsanity.com/paper/2506.04838/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 5 of 5)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [SecureBERT: A Domain-Specific Language Model for Cybersecurity](https://mlsanity.com/paper/2204.02685/) (2022)
- [ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking System](https://mlsanity.com/paper/2411.17058/) (2024)
- [Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models](https://mlsanity.com/paper/2403.00878/) (2024)

## Abstract (arXiv)

The complexity of modern computing environments and the growing sophistication of cyber threats necessitate a more robust, adaptive, and automated approach to security enforcement. In this paper, we present a framework leveraging large language models (LLMs) for automating attack mitigation policy compliance through an innovative combination of in-context learning and retrieval-augmented generation (RAG). We begin by describing how our system collects and manages both tool and API specifications, storing them in a vector database to enable efficient retrieval of relevant information. We then detail the architectural pipeline that first decomposes high-level mitigation policies into discrete tasks and subsequently translates each task into a set of actionable API calls. Our empirical evaluation, conducted using publicly available CTI policies in STIXv2 format and Windows API documentation, demonstrates significant improvements in precision, recall, and F1-score when employing RAG compared to a non-RAG baseline.