mlsanityData to 14 Oct 2026

Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety

arXiv 2604.12311 ↗First posted 14 Apr 2026cs.SE cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

The emergence of vibe coding, a paradigm where non-technical users instruct Large Language Models (LLMs) to generate executable codes via natural language, presents both significant opportunities and severe risks for the construction industry. While empowering construction personnel such as the safety managers, foremen, and workers to develop tools and software, the probabilistic nature of LLMs introduces the threat of silent failures, wherein generated code compiles perfectly but executes flawed mathematical safety logic. This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts generated by three frontier models, Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. Utilizing a persona-driven prompt dataset (n=150) and a bifurcated evaluation pipeline comprising isolated dynamic sandboxing and an LLM-as-a-Judge, the research quantifies the severe limits of zero-shot vibe codes for construction safety. The findings reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables. Furthermore, while the models demonstrated high foundational execution viability (~85%), this syntactic reliability actively masked logic deficits and a severe lack of defensive programming. Among successfully executed scripts, the study identified an alarming ~45% overall Silent Failure Rate, with GPT-4o-Mini generating mathematically inaccurate outputs in ~56% of its functional code. The results demonstrate that current LLMs lack the deterministic rigor required for standalone safety engineering, necessitating the adoption of deterministic AI wrappers and strict governance for cyber-physical deployments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

31 of 58 references.

References · 31

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2024A Survey on Large Language Models for Code Generationcited by 658
  4. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  5. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 107
  6. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  7. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  8. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  9. 2024Automatic Programming: Large Language Models and Beyondcited by 36
  10. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  11. 2024Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Modelscited by 30
  12. 2025Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Reviewcited by 30
19 more
  1. 2025Vibe coding: programming through conversation with artificial intelligencecited by 29
  2. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 27
  3. 2025A Survey of Vibe Coding with Large Language Modelscited by 27
  4. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 21
  5. 2025Vibe Coding as a Reconfiguration of Intent Mediation in Software Development: Definition, Implications, and Research Agendacited by 13
  6. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 10
  7. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  8. 2026ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimizationcited by 8
  9. 2024Security of Language Models for Code: A Systematic Literature Reviewcited by 7
  10. 2025From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systemscited by 7
  11. 2025The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Modelscited by 7
  12. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  13. 2025Building Software by Rolling the Dice: A Qualitative Study of Vibe Codingcited by 5
  14. 2025Automated Validation of LLM-based Evaluators for Software Engineering Artifactscited by 4
  15. 2025Review of Tools for Zero-Code LLM Based Application Developmentcited by 4
  16. 2025LLM4FaaS: No-Code Application Development using LLMs and FaaScited by 2
  17. 2025Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Datacited by 2
  18. 2025On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CIcited by 2
  19. 2025Taming Silent Failures: A Framework for Verifiable AI Reliabilitycited by 1

Built on by

Not cited yet.