# Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation

arXiv: https://arxiv.org/abs/2506.05623 · HTML: https://mlsanity.com/paper/2506.05623/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 91 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Leveraging Automated Unit Tests for Unsupervised Code Translation](https://mlsanity.com/paper/2110.06773/) (2021)
- [Generating Natural Language Proofs with Verifier-Guided Search](https://mlsanity.com/paper/2205.12443/) (2022)
- [A Performance Study of LLM-Generated Code on Leetcode](https://mlsanity.com/paper/2407.21579/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback](https://mlsanity.com/paper/2403.16792/) (2024)
- [Human-in-the-loop Artificial Intelligence](https://mlsanity.com/paper/1710.08191/) (2017)
- [CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging](https://mlsanity.com/paper/2502.05664/) (2025)
- [LLM A*: Human in the Loop Large Language Models Enabled A* Search for Robotics](https://mlsanity.com/paper/2312.01797/) (2023)
- [Bridging Design and Development with Automated Declarative UI Code Generation](https://mlsanity.com/paper/2409.11667/) (2024)
- [A Survey of using Large Language Models for Generating Infrastructure as Code](https://mlsanity.com/paper/2404.00227/) (2024)
- [Analyzing and Mitigating (with LLMs) the Security Misconfigurations of Helm Charts from Artifact Hub](https://mlsanity.com/paper/2403.09537/) (2024)
- [Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlays](https://mlsanity.com/paper/2210.07259/) (2022)
- [A Large-scale Investigation of Semantically Incompatible APIs behind Compatibility Issues in Android Apps](https://mlsanity.com/paper/2406.17431/) (2024)
- [Using a Feedback Loop for LLM-based Infrastructure as Code Generation](https://mlsanity.com/paper/2411.19043/) (2024)
- [Privacy Bills of Materials: A Transparent Privacy Information Inventory for Collaborative Privacy Notice Generation in Mobile App Development](https://mlsanity.com/paper/2501.01131/) (2025)
- [Mining Android API Usage to Generate Unit Test Cases for Pinpointing Compatibility Issues](https://mlsanity.com/paper/2208.13417/) (2022)
- [Taming Android Fragmentation through Lightweight Crowdsourced Testing](https://mlsanity.com/paper/2304.04347/) (2023)
- [Do Chase Your Tail! Missing Key Aspects Augmentation in Textual Vulnerability Descriptions of Long-tail Software through Feature Inference](https://mlsanity.com/paper/2405.07430/) (2024)
- [A First Look at CI/CD Adoptions in Open-Source Android Apps](https://mlsanity.com/paper/2210.12581/) (2022)
- [A Solution toward Transparent and Practical AI Regulation: Privacy Nutrition Labels for Open-source Generative AI-based Applications](https://mlsanity.com/paper/2407.15407/) (2024)

## Built on by (most cited first, 1 of 1)

- [Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code](https://mlsanity.com/paper/2605.09360/) (2026)

## Abstract (arXiv)

Infrastructure-as-Code (IaC) generation holds significant promise for automating cloud infrastructure provisioning. Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development by generating deployable infrastructure templates from natural language descriptions. However, current evaluation focuses on syntactic correctness while ignoring deployability, the critical measure of the utility of IaC configuration files. Six state-of-the-art LLMs performed poorly on deployability, achieving only 20.8~30.2% deployment success rate on the first attempt. In this paper, we construct DPIaC-Eval, the first deployability-centric IaC template benchmark consisting of 153 real-world scenarios cross 58 unique services. Also, we propose an LLM-based deployability-centric framework, dubbed IaCGen, that uses iterative feedback mechanism encompassing format verification, syntax checking, and live deployment stages, thereby closely mirroring the real DevOps workflows. Results show that IaCGen can make 54.6~91.6% generated IaC templates from all evaluated models deployable in the first 10 iterations. Additionally, human-in-the-loop feedback that provide direct guidance for the deployability errors, can further boost the performance to over 90% passItr@25 on all evaluated LLMs. Furthermore, we explore the trustworthiness of the generated IaC templates on user intent alignment and security compliance. The poor performance (25.2% user requirement coverage and 8.4% security compliance rate) indicates a critical need for continued research in this domain.