# Using LLMs in Generating Design Rationale for Software Architecture Decisions

arXiv: https://arxiv.org/abs/2504.20781 · HTML: https://mlsanity.com/paper/2504.20781/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 71 references that are in the map.
- First posted on arXiv 29 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives](https://mlsanity.com/paper/2310.01152/) (2023)
- [Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions](https://mlsanity.com/paper/2308.02312/) (2023)
- [Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents](https://mlsanity.com/paper/2405.10467/) (2024)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [MARE: Multi-Agents Collaboration Framework for Requirements Engineering](https://mlsanity.com/paper/2405.03256/) (2024)
- [Measuring Personalization of Web Search](https://mlsanity.com/paper/1706.05011/) (2017)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [How Effective are Large Language Models in Generating Software Specifications?](https://mlsanity.com/paper/2306.03324/) (2023)
- [Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction](https://mlsanity.com/paper/2311.04532/) (2023)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [Using Large Language Models for Commit Message Generation: A Preliminary Study](https://mlsanity.com/paper/2401.05926/) (2024)
- [Recommending Stack Overflow Posts for Fixing Runtime Exceptions using Failure Scenario Matching](https://mlsanity.com/paper/2009.10174/) (2020)
- [Do Code LLMs Understand Design Patterns?](https://mlsanity.com/paper/2501.04835/) (2025)
- [Addressing Compiler Errors: Stack Overflow or Large Language Models?](https://mlsanity.com/paper/2307.10793/) (2023)
- [DRAFT-ing Architectural Design Decisions using LLMs](https://mlsanity.com/paper/2504.08207/) (2025)
- [Characterizing Architecture Related Posts and Their Usefulness in Stack Overflow](https://mlsanity.com/paper/2301.00943/) (2023)
- [Evaluating Privacy Questions From Stack Overflow: Can ChatGPT Compete?](https://mlsanity.com/paper/2306.11174/) (2023)
- [S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document](https://mlsanity.com/paper/2403.10588/) (2024)

## Built on by (most cited first, 4 of 4)

- [ArchAgent: Scalable Legacy Software Architecture Recovery with LLMs](https://mlsanity.com/paper/2601.13007/) (2026)
- [(How) Do Large Language Models Understand High-Level Message Sequence Charts?](https://mlsanity.com/paper/2605.13773/) (2026)
- [Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey](https://mlsanity.com/paper/2605.01392/) (2026)
- [Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study](https://mlsanity.com/paper/2609.03721/) (2026)

## Abstract (arXiv)

Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Additionally, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.