# Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study

arXiv: https://arxiv.org/abs/2403.01709 · HTML: https://mlsanity.com/paper/2403.01709/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks](https://mlsanity.com/paper/2102.12592/) (2021)
- [Assemble Foundation Models for Automatic Code Summarization](https://mlsanity.com/paper/2201.05222/) (2022)
- [Developer-Intent Driven Code Comment Generation](https://mlsanity.com/paper/2302.07055/) (2023)

## Built on by (most cited first, 19 of 19)

- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [Generative AI for Software Architecture. Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2503.13310/) (2025)
- [Knowledge-Based Multi-Agent Framework for Automated Software Architecture Design](https://mlsanity.com/paper/2503.20536/) (2025)
- [Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation](https://mlsanity.com/paper/2508.14273/) (2025)
- [Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithm](https://mlsanity.com/paper/2408.11198/) (2024)
- [DRAFT-ing Architectural Design Decisions using LLMs](https://mlsanity.com/paper/2504.08207/) (2025)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [A Multi-Agent Framework Integrating Large Language Models and Generative AI for Accelerated Metamaterial Design](https://mlsanity.com/paper/2503.19889/) (2025)
- [From Requirements to Architecture: Semi-Automatically Generating Software Architectures](https://mlsanity.com/paper/2504.12192/) (2025)
- [MAAD: Automate Software Architecture Design through Knowledge-Driven Multi-Agent Collaboration](https://mlsanity.com/paper/2507.21382/) (2025)
- [Evaluating the effectiveness of LLM-based interoperability](https://mlsanity.com/paper/2510.23893/) (2025)
- [Evaluating Large Language Models for Detecting Architectural Decision Violations](https://mlsanity.com/paper/2602.07609/) (2026)
- [LLM-based Automated Architecture View Generation: Where Are We Now?](https://mlsanity.com/paper/2603.21178/) (2026)
- [Toward a Sustainable Software Architecture Community: Evaluating ICSA's Environmental Impact](https://mlsanity.com/paper/2604.04096/) (2026)
- [Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey](https://mlsanity.com/paper/2605.01392/) (2026)
- [Bridging Requirements and Architecture: Multi-Agent Orchestration with External Knowledge and Hierarchical Memory](https://mlsanity.com/paper/2606.01385/) (2026)
- [From Textual Requirements to Microservice Architectures - A Comprehensive Evaluation of LLM-Based Design Synthesis](https://mlsanity.com/paper/2607.28307/) (2026)
- [GADR: Gathering Architecture Decision Records from Meeting Transcriptions](https://mlsanity.com/paper/2608.17694/) (2026)
- [Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study](https://mlsanity.com/paper/2609.03721/) (2026)

## Abstract (arXiv)

Architectural Knowledge Management (AKM) involves the organized handling of information related to architectural decisions and design within a project or organization. An essential artifact of AKM is the Architecture Decision Records (ADR), which documents key design decisions. ADRs are documents that capture decision context, decision made and various aspects related to a design decision, thereby promoting transparency, collaboration, and understanding. Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation. However, the effectiveness of LLM for ADR generation or understanding is something that has not been explored. To this end, in this work, we perform an exploratory study that aims to investigate the feasibility of using LLM for the generation of ADRs given the decision context. In our exploratory study, we utilize GPT and T5-based models with 0-shot, few-shot, and fine-tuning approaches to generate the Decision of an ADR given its Context. Our results indicate that in a 0-shot setting, state-of-the-art models such as GPT-4 generate relevant and accurate Design Decisions, although they fall short of human-level performance. Additionally, we observe that more cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot setting, and smaller models such as Flan-T5 can yield comparable results after fine-tuning. To conclude, this exploratory study suggests that LLM can generate Design Decisions, but further research is required to attain human-level generation and establish standardized widespread adoption.