# Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

arXiv: https://arxiv.org/abs/2508.11824 · HTML: https://mlsanity.com/paper/2508.11824/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 17 of 17)

- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion](https://mlsanity.com/paper/2406.09834/) (2024)
- [Supersonic: Learning to Generate Source Code Optimizations in C/C++](https://mlsanity.com/paper/2309.14846/) (2023)
- [Model Editing for LLMs4Code: How Far are We?](https://mlsanity.com/paper/2411.06638/) (2024)
- [An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues](https://mlsanity.com/paper/2403.10468/) (2024)
- [Faster Configuration Performance Bug Testing with Neural Dual-level Prioritization](https://mlsanity.com/paper/2501.15392/) (2025)
- [Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks](https://mlsanity.com/paper/2408.15207/) (2024)
- [Knowledge-Enhanced Program Repair for Data Science Code](https://mlsanity.com/paper/2502.09771/) (2025)
- [TRACE: Transparent Web Reliability Assessment with Contextual Explanations](https://mlsanity.com/paper/2506.12072/) (2025)
- [Diversity Drives Fairness: Ensemble of Higher Order Mutants for Intersectional Fairness of Machine Learning Software](https://mlsanity.com/paper/2412.08167/) (2024)
- [FairSense: Long-Term Fairness Analysis of ML-Enabled Systems](https://mlsanity.com/paper/2501.01665/) (2025)
- [SparseCoder: Advancing Source Code Analysis with Sparse Attention and Learned Token Pruning](https://mlsanity.com/paper/2310.07109/) (2023)
- [Diversity's Double-Edged Sword: Analyzing Race's Effect on Remote Pair Programming Interactions](https://mlsanity.com/paper/2404.07427/) (2024)
- [Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets](https://mlsanity.com/paper/2406.17651/) (2024)
- [Investigating the Impact of Interpersonal Challenges on Feeling Welcome in OSS](https://mlsanity.com/paper/2411.01601/) (2024)
- [User Personas Improve Social Sustainability by Encouraging Software Developers to Deprioritize Antisocial Features](https://mlsanity.com/paper/2412.10672/) (2024)
- [Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection](https://mlsanity.com/paper/2502.18883/) (2025)

## Built on by (most cited first, 4 of 4)

- [Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering](https://mlsanity.com/paper/2601.14470/) (2026)
- [PIE: Performance Interval Estimation for Free-Form Generation Tasks](https://mlsanity.com/paper/2509.07309/) (2025)
- [Governance Controls for AI-Generated Test Artifacts in Autonomous Software Testing](https://mlsanity.com/paper/2606.08806/) (2026)
- [On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models](https://mlsanity.com/paper/2608.10530/) (2026)

## Abstract (arXiv)

The integration of Large Language Models (LLMs) into software engineering has revolutionized code generation, enabling unprecedented productivity through promptware and autonomous AI agents. However, this transformation introduces significant risks, including insecure code generation, hallucinated outputs, irreversible actions, and a lack of transparency and accountability. Incidents like the Replit database deletion underscore the urgent need for robust safety and governance mechanisms. This paper comprehensively analyzes the inherent challenges of LLM-assisted code generation, such as vulnerability inheritance, overtrust, misinterpretation, and the absence of standardized validation and rollback protocols. To address these, we propose the SAFE-AI Framework, a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability. The framework integrates guardrails, sandboxing, runtime verification, risk-aware logging, human-in-the-loop systems, and explainable AI techniques to mitigate risks while fostering trust and compliance. We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight. Additionally, we identify open problems, including the lack of standardized benchmarks for code specific hallucinations and autonomy levels, and propose future research directions for hybrid verification, semantic guardrails, and proactive governance tools. Through detailed comparisons of autonomy control, prompt engineering, explainability, and governance frameworks, this paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA to ensure safe, transparent, and accountable AI-driven development.