mlsanityData to 14 Oct 2026

Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

arXiv 2607.20478 ↗First posted 29 May 2026cs.SE cs.AIcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy constraints, not merely producing syntactically plausible configurations. We present a verifier-first empirical study of seven agentic strategies for Terraform generation evaluated on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 intent policies. Our evaluation separates failures into three verifier stages (terraform validate, terraform plan, opa eval) and applies McNemar's test with Wilson confidence intervals on all pairwise comparisons (n=186, alpha=0.05). We report five principal findings. (1) Active retrieval via ReAct agents with MCP or ChromaDB-backed RAG raises Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001), primarily by reducing VALIDATE_FAIL from 144 to 66 tasks. (2) Iterative refinement with verifier feedback achieves 62.9% (Qwen 7B) and 84.4% (GPT-4o) pass@1, exhibiting binary convergence -- tasks either resolve in one retry or exhaust the budget. (3) GEPA reflective instruction optimization raises the Active RAG baseline by +7.5 pp (p=0.026) using only 80 verifier-guided rollouts, providing evidence that prompt optimizers can improve verifiable IaC generation without weight updates. (4) SIMBA teacher-free demonstration injection achieves performance equivalent to Active RAG (p=1.0) without retrieval infrastructure, but fails to address the dominant SELF_DEFINED_PROPERTY error class (50% of failures). (5) A diagnostic Rego-injection experiment shows that 79% of post-refinement OPA failures are information-gap failures resolvable when policy text is visible (p=0.016), motivating policy

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 0 of its 12 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

0 of 12 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Code-Orchestrated Generation for Reliable Infrastructure-as-Code
  2. 2026TerraFormer: Automated Infrastructure-as-Code with LLMs Fine-Tuned via Policy-Guided Verifier Feedback
  3. 2025IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
  4. 2026TerraRepair: A Tool-Grounded LLM Agent for Infrastructure-as-Code Repair
  5. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
  6. 2024Using a Feedback Loop for LLM-based Infrastructure as Code Generation
  7. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review
  8. 2025Sherlock: Reliable and Efficient Agentic Workflow Execution