# Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey

arXiv: https://arxiv.org/abs/2605.01392 · HTML: https://mlsanity.com/paper/2605.01392/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 2 May 2026.

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://mlsanity.com/paper/2507.09089/) (2025)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [Requirements Engineering using Generative AI: Prompts and Prompting Patterns](https://mlsanity.com/paper/2311.03832/) (2023)
- [Software Architecture Meets LLMs: A Systematic Literature Review](https://mlsanity.com/paper/2505.16697/) (2025)
- [LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World](https://mlsanity.com/paper/2502.02539/) (2025)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [NOMAD: A Multi-Agent LLM System for UML Class Diagram Generation from Natural Language Requirements](https://mlsanity.com/paper/2511.22409/) (2025)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across software engineering tasks, including software design, an area traditionally regarded as highly dependent on human expertise and judgment. However, limited research has examined how LLMs are used in software design, aswell as the associated benefits and drawbacks. This paper addresses this gap by empirically investigating how software developers use LLMs in software design. We conducted a mixed-methods study, combining a mining study of 291 developer-ChatGPT conversations shared on GitHub with a survey of 65 software practitioners. From the mined conversations, we identified nine categories of design tasks supported by ChatGPT, including architecture design, data model design, and the use of design patterns.We further characterize developer-ChatGPT interactions, showing that developers in the mined conversations primarily use ChatGPT for knowledge acquisition and designrelated code generation, with most tasks situated at the detailed design level. The survey participants reported seven key benefits, such as better technology selection and early detection of design flaws, and six limitations, including lengthy outputs, inexecutable or incorrect code, and dependence on project context. These findings provide an evidence-based characterization of current LLM use in software design from both open-source and practitioner perspectives.