# Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

arXiv: https://arxiv.org/abs/2608.02672 · HTML: https://mlsanity.com/paper/2608.02672/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 2 Aug 2026.

## Builds on (most cited first, 9 of 9)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [GLITCH: Automated Polyglot Security Smell Detection in Infrastructure as Code](https://mlsanity.com/paper/2205.14371/) (2022)
- [TerraFormer: Automated Infrastructure-as-Code with LLMs Fine-Tuned via Policy-Guided Verifier Feedback](https://mlsanity.com/paper/2601.08734/) (2026)
- [Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats](https://mlsanity.com/paper/2509.05303/) (2025)
- [IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection](https://mlsanity.com/paper/2512.14792/) (2025)

## Abstract (arXiv)

Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code is an open question. We benchmark seven models, three closed LLMs (Claude Opus 4, GPT-5.4, Gemini 2.5 Pro) and four open SLMs (Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, Magicoder-S-CL-7B), on AWS Terraform generation across 17 scenarios, integrating Checkov and Trivy scanners into a GitLab CI/CD pipeline and evaluating two prompt strategies at three security levels (pass@5). Syntactic validity and security compliance are largely orthogonal properties in LLM-generated IaC, a model that reliably produces well-formed Terraform does not necessarily produce secure Terraform: WizardCoder-33B achieves 77.8% validate rate yet zero Checkov compliance, while Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates under detailed security prompting. Consequently, prompt engineering alone is insufficient: automated multi-tool scanning remains a necessary complement to LLM-assisted IaC generation regardless of model family or prompt strategy. All artifacts are publicly available.