# Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety

arXiv: https://arxiv.org/abs/2604.12311 · HTML: https://mlsanity.com/paper/2604.12311/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation](https://mlsanity.com/paper/2502.12468/) (2025)
- [Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models](https://mlsanity.com/paper/2406.08731/) (2024)
- [Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review](https://mlsanity.com/paper/2510.00328/) (2025)
- [Vibe coding: programming through conversation with artificial intelligence](https://mlsanity.com/paper/2506.23253/) (2025)
- [Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking](https://mlsanity.com/paper/2407.02395/) (2024)
- [A Survey of Vibe Coding with Large Language Models](https://mlsanity.com/paper/2510.12399/) (2025)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [Vibe Coding as a Reconfiguration of Intent Mediation in Software Development: Definition, Implications, and Research Agenda](https://mlsanity.com/paper/2507.21928/) (2025)
- [A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?](https://mlsanity.com/paper/2411.01414/) (2024)
- [SandboxEval: Towards Securing Test Environment for Untrusted Code](https://mlsanity.com/paper/2504.00018/) (2025)
- [ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization](https://mlsanity.com/paper/2602.15983/) (2026)
- [Security of Language Models for Code: A Systematic Literature Review](https://mlsanity.com/paper/2410.15631/) (2024)
- [From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems](https://mlsanity.com/paper/2502.07974/) (2025)
- [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://mlsanity.com/paper/2504.20612/) (2025)
- [Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications](https://mlsanity.com/paper/2508.12358/) (2025)
- [Building Software by Rolling the Dice: A Qualitative Study of Vibe Coding](https://mlsanity.com/paper/2512.22418/) (2025)
- [Automated Validation of LLM-based Evaluators for Software Engineering Artifacts](https://mlsanity.com/paper/2508.02827/) (2025)
- [Review of Tools for Zero-Code LLM Based Application Development](https://mlsanity.com/paper/2510.19747/) (2025)
- [LLM4FaaS: No-Code Application Development using LLMs and FaaS](https://mlsanity.com/paper/2502.14450/) (2025)
- [Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data](https://mlsanity.com/paper/2508.06484/) (2025)
- [On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI](https://mlsanity.com/paper/2509.14347/) (2025)

## Abstract (arXiv)

The emergence of vibe coding, a paradigm where non-technical users instruct Large Language Models (LLMs) to generate executable codes via natural language, presents both significant opportunities and severe risks for the construction industry. While empowering construction personnel such as the safety managers, foremen, and workers to develop tools and software, the probabilistic nature of LLMs introduces the threat of silent failures, wherein generated code compiles perfectly but executes flawed mathematical safety logic. This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts generated by three frontier models, Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. Utilizing a persona-driven prompt dataset (n=150) and a bifurcated evaluation pipeline comprising isolated dynamic sandboxing and an LLM-as-a-Judge, the research quantifies the severe limits of zero-shot vibe codes for construction safety. The findings reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables. Furthermore, while the models demonstrated high foundational execution viability (~85%), this syntactic reliability actively masked logic deficits and a severe lack of defensive programming. Among successfully executed scripts, the study identified an alarming ~45% overall Silent Failure Rate, with GPT-4o-Mini generating mathematically inaccurate outputs in ~56% of its functional code. The results demonstrate that current LLMs lack the deterministic rigor required for standalone safety engineering, necessitating the adoption of deterministic AI wrappers and strict governance for cyber-physical deployments.