# Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

arXiv: https://arxiv.org/abs/2608.28021 · HTML: https://mlsanity.com/paper/2608.28021/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [Where Are The Gaps? A Systematic Mapping Study of Infrastructure as Code Research](https://mlsanity.com/paper/1807.04872/) (2018)
- [GLITCH: Automated Polyglot Security Smell Detection in Infrastructure as Code](https://mlsanity.com/paper/2205.14371/) (2022)
- [Exploring Security Practices in Infrastructure as Code: An Empirical Study](https://mlsanity.com/paper/2308.03952/) (2023)

## Abstract (arXiv)

Large language models increasingly author Infrastructure-as-Code (IaC), where one insecure default is provisioned straight into production. Prior evaluations report vulnerability counts for models only, and so cannot say whether models are worse than the engineers they assist. We present GenIaC-SecBench: 100 deployment scenarios across 12 model configurations from six vendors, open and closed weights, yielding 1,196 artifacts scanned by three policy engines (Checkov, Trivy, KICS) at complete coverage. Crucially we scan 634 human-authored IaC templates with the identical toolchain, giving the first size-matched human security baseline for this task. Vulnerability density is strongly inverse to artifact size (Spearman ρ=-0.55, p<10-77), so unmatched comparisons measure size, not security. Size-matched, every configuration exceeds the human baseline at 3.21× to 3.87×, and the gap widens as tasks get simpler (4.9× at one resource, 1.4× at twenty or more). A majority of scenarios prescribe a security state rather than specifying function alone, so we stratify by prompt class: pooled the gap is 3.50×, and excluding every scenario that explicitly requests an insecure configuration still leaves all configurations above baseline (2.4× to 4.2×). The corpus cannot isolate unprompted default posture, and we say so. Decomposing "reasoning" into standard generation, prompted chain-of-thought, and vendor extended-thinking APIs, extended thinking beats prompted CoT (-12.0%, p=0.0013) while prompted CoT alone is indistinguishable from standard (-1.3%, n.s.); it consumes under 1% of the output budget, bounding the effect. Two negative results: more deployable models are not more vulnerable (r=0.158, p=0.625), and complete-case Friedman is uncomputable here, motivating Skillings-Mack. All code and data are released.