mlsanityData to 7 Oct 2026

Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

arXiv 2608.28021 ↗First posted 28 Aug 2026cs.CR cs.AI cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models increasingly author Infrastructure-as-Code (IaC), where one insecure default is provisioned straight into production. Prior evaluations report vulnerability counts for models only, and so cannot say whether models are worse than the engineers they assist. We present GenIaC-SecBench: 100 deployment scenarios across 12 model configurations from six vendors, open and closed weights, yielding 1,196 artifacts scanned by three policy engines (Checkov, Trivy, KICS) at complete coverage. Crucially we scan 634 human-authored IaC templates with the identical toolchain, giving the first size-matched human security baseline for this task. Vulnerability density is strongly inverse to artifact size (Spearman ρ=-0.55, p<10-77), so unmatched comparisons measure size, not security. Size-matched, every configuration exceeds the human baseline at 3.21× to 3.87×, and the gap widens as tasks get simpler (4.9× at one resource, 1.4× at twenty or more). A majority of scenarios prescribe a security state rather than specifying function alone, so we stratify by prompt class: pooled the gap is 3.50×, and excluding every scenario that explicitly requests an insecure configuration still leaves all configurations above baseline (2.4× to 4.2×). The corpus cannot isolate unprompted default posture, and we say so. Decomposing "reasoning" into standard generation, prompted chain-of-thought, and vendor extended-thinking APIs, extended thinking beats prompted CoT (-12.0%, p=0.0013) while prompted CoT alone is indistinguishable from standard (-1.3%, n.s.); it consumes under 1% of the output budget, bounding the effect. Two negative results: more deployable models are not more vulnerable (r=0.158, p=0.625), and complete-case Friedman is uncomputable here, motivating Skillings-Mack. All code and data are released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 48 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

14 of 48 references.

References · 14

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  9. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  10. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 87
  11. 2024How secure is AI-generated Code: A Large-Scale Comparison of Large Language Modelscited by 23
  12. 2018Where Are The Gaps? A Systematic Mapping Study of Infrastructure as Code Researchcited by 9
2 more
  1. 2022GLITCH: Automated Polyglot Security Smell Detection in Infrastructure as Codecited by 5
  2. 2023Exploring Security Practices in Infrastructure as Code: An Empirical Studycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection
  2. 2025Automated Vulnerability Validation and Verification: A Large Language Model Approach
  3. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  4. 2026RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
  5. 2026An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods
  6. 2026Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
  7. 2026SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code ↗
  8. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?