# Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report arXiv: https://arxiv.org/abs/2608.11965 ยท HTML: https://mlsanity.com/paper/2608.11965/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 29 of its 79 references that are in the map. - First posted on arXiv 12 Aug 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023) - [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023) - [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023) - [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024) - [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024) - [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024) - [Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues](https://mlsanity.com/paper/2307.12596/) (2023) - [Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks](https://mlsanity.com/paper/2102.02017/) (2021) - [Guidelines for including grey literature and conducting multivocal literature reviews in software engineering](https://mlsanity.com/paper/1707.02553/) (2017) - [Investigating Gender Bias in BERT](https://mlsanity.com/paper/2009.05021/) (2020) - [Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective](https://mlsanity.com/paper/2411.14258/) (2024) - [Understanding the Factors that Impact the Popularity of GitHub Repositories](https://mlsanity.com/paper/1606.04984/) (2016) - [Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models](https://mlsanity.com/paper/2312.05562/) (2023) - [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018) - [Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study](https://mlsanity.com/paper/2305.11164/) (2023) - [Requirements are All You Need: From Requirements to Code with LLMs](https://mlsanity.com/paper/2406.10101/) (2024) - [Guidelines for Empirical Studies in Software Engineering involving Large Language Models](https://mlsanity.com/paper/2508.15503/) (2025) - [Autonomous Agents in Software Development: A Vision Paper](https://mlsanity.com/paper/2311.18440/) (2023) - [A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama](https://mlsanity.com/paper/2405.03616/) (2024) - [Prompt engineering and its implications on the energy consumption of Large Language Models](https://mlsanity.com/paper/2501.05899/) (2025) - [Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarization](https://mlsanity.com/paper/2503.10876/) (2025) - [Automatically Categorising GitHub Repositories by Application Domain](https://mlsanity.com/paper/2208.00269/) (2022) - [Qualitative Research Methods in Software Engineering: Past, Present, and Future](https://mlsanity.com/paper/2502.07220/) (2025) - [Many Hands Make Light Work: An LLM-based Multi-Agent System for Detecting Malicious PyPI Packages](https://mlsanity.com/paper/2601.12148/) (2026) - [Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach](https://mlsanity.com/paper/2606.24689/) (2026) ## Built on by (most cited first, 1 of 1) - [AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering](https://mlsanity.com/paper/2609.32662/) (2026) ## Abstract (arXiv) The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. First, we conducted a quantitative analysis of the most relevant open source MAS frameworks by evaluating their documentation, features, and capabilities from the developers' perspective. Second, we performed a qualitative evaluation of a subset of the selected frameworks by implementing a common use case: the summarization of README.MD files. The findings show that the selected frameworks provide a good coverage of fundamental components of MAS, though advanced features such as telemetry of agents are still missing. In addition, the empirical evaluation shows that there is no significant difference in terms of ROUGE scores considering the summarization task. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.