# DRAFT-ing Architectural Design Decisions using LLMs

arXiv: https://arxiv.org/abs/2504.08207 · HTML: https://mlsanity.com/paper/2504.08207/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [The Current Challenges of Software Engineering in the Era of Large Language Models](https://mlsanity.com/paper/2412.14554/) (2024)
- [Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges](https://mlsanity.com/paper/2403.00784/) (2024)
- [Retrieval-augmented Few-shot Medical Image Segmentation with Foundation Models](https://mlsanity.com/paper/2408.08813/) (2024)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization](https://mlsanity.com/paper/2410.02721/) (2024)
- [Generative AI for Requirements Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2409.06741/) (2024)
- [From Requirements to Architecture: An AI-Based Journey to Semi-Automatically Generate Software Architectures](https://mlsanity.com/paper/2401.14079/) (2024)
- [Exploring Large Language Models for Code Explanation](https://mlsanity.com/paper/2310.16673/) (2023)
- [Archify: A Recommender System of Architectural Design Decisions](https://mlsanity.com/paper/2106.08115/) (2021)
- [Teaching Dense Retrieval Models to Specialize with Listwise Distillation and LLM Data Augmentation](https://mlsanity.com/paper/2502.19712/) (2025)
- [Researcher Bias in Software Engineering Experiments: a Qualitative Investigation](https://mlsanity.com/paper/2008.12528/) (2020)
- [SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models](https://mlsanity.com/paper/2502.18168/) (2025)

## Built on by (most cited first, 4 of 4)

- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [Evaluating Large Language Models for Detecting Architectural Decision Violations](https://mlsanity.com/paper/2602.07609/) (2026)
- [Automating Domain-Driven Design: Experience with a Prompting Framework](https://mlsanity.com/paper/2603.26244/) (2026)
- [GADR: Gathering Architecture Decision Records from Meeting Transcriptions](https://mlsanity.com/paper/2608.17694/) (2026)

## Abstract (arXiv)

Architectural Knowledge Management (AKM) is crucial for software development but remains challenging due to the lack of standardization and high manual effort. Architecture Decision Records (ADRs) provide a structured approach to capture Architecture Design Decisions (ADDs), but their adoption is limited due to the manual effort involved and insufficient tool support. Our previous work has shown that Large Language Models (LLMs) can assist in generating ADDs. However, simply prompting the LLM does not produce quality ADDs. Moreover, using third-party LLMs raises privacy concerns, while self-hosting them poses resource challenges. To this end, we experimented with different approaches like few-shot, retrieval-augmented generation (RAG) and fine-tuning to enhance LLM's ability to generate ADDs. Our results show that both techniques improve effectiveness. Building on this, we propose Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, which combines the strengths of all these three approaches for more effective ADD generation. DRAFT operates in two phases: an offline phase that fine-tunes an LLM on generating ADDs augmented with retrieved examples and an online phase that generates ADDs by leveraging retrieved ADRs and the fine-tuned model. We evaluated DRAFT against existing approaches on a dataset of 4,911 ADRs and various LLMs and analyzed them using automated metrics and human evaluations. Results show DRAFT outperforms all other approaches in effectiveness while maintaining efficiency. Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.