# Evaluating Large Language Models for Detecting Architectural Decision Violations

arXiv: https://arxiv.org/abs/2602.07609 · HTML: https://mlsanity.com/paper/2602.07609/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [Generative AI for Software Architecture. Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2503.13310/) (2025)
- [Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study](https://mlsanity.com/paper/2403.15756/) (2024)
- [Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis](https://mlsanity.com/paper/2406.10273/) (2024)
- [On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?](https://mlsanity.com/paper/2412.11698/) (2024)
- [LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World](https://mlsanity.com/paper/2502.02539/) (2025)
- [DRAFT-ing Architectural Design Decisions using LLMs](https://mlsanity.com/paper/2504.08207/) (2025)
- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [Survey on Tools and Techniques Detecting Microservice API Patterns](https://mlsanity.com/paper/2205.10133/) (2022)
- [Emerging Trends in Software Architecture from the Practitioners Perspective: A Five Year Review](https://mlsanity.com/paper/2507.14554/) (2025)
- [AI Techniques in the Microservices Life-Cycle: A Systematic Mapping Study](https://mlsanity.com/paper/2305.16092/) (2023)

## Abstract (arXiv)

Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.