The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems
On the map: Retrieval-augmented generation → RAG for code
also in 1 more
Abstract (arXiv)
Large Language Models (LLMs) are increasingly utilized to automate several software engineering tasks, including code completion, code summarization, testing, and the generation of repository-level documentation. While Multi-Agent Systems (MAS) are often adopted to support such tasks under the premise that task decomposition improves performance, the impact of architectural complexity on practical efficiency remains under-examined. This study empirically evaluates Retrieval-Augmented Generation (RAG) dependent architectures for the generation of README files for GitHub repositories. In this work, we conducted a systematic comparison between a Single-Agent pipeline, a specialized MAS, and a developer-guided planning (DevPlan) variant, benchmarked against LARCH -- a state-of-the-art baseline -- and the original ground truth. Results indicate a critical architectural trade-off: the Single-Agent pipeline achieves lexical quality comparable to MAS while reducing token consumption by 86% and operating at twice the speed. In contrast, manual taxonomy analysis demonstrates that MAS achieves high structural consistency (98%), resolving formatting issues observed in single-agent approaches. Autonomous planning is identified as the primary pipeline bottleneck; incorporating lightweight developer-guided plans produces the highest overall documentation quality, surpassing all the analyzed configurations.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 36 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
27 of 36 references.
References · 27
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
- 2024A Survey on Large Language Models for Code Generationcited by 646
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2023Using an LLM to Help With Code Understandingcited by 222
- 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
- 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
- 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
15 more
- 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 63
- 2022Automatic Code Documentation Generation Using GPT-3cited by 37
- 2025A Survey on Data Contamination for Large Language Modelscited by 33
- 2025Single-agent or Multi-agent Systems? Why Not Both?cited by 28
- 2016Understanding the Factors that Impact the Popularity of GitHub Repositoriescited by 27
- 2023A Comparative Analysis of Large Language Models for Code Documentation Generationcited by 24
- 2025DocAgent: A Multi-Agent System for Automated Code Documentation Generationcited by 16
- 2024Free and Customizable Code Documentation with LLMs: A Fine-Tuning Approachcited by 10
- 2025Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarizationcited by 4
- 2025CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebasescited by 4
- 2024Context-aware Code Summary Generationcited by 3
- 2025RepoSummary: Feature-Oriented Summarization and Documentation Generation for Code Repositoriescited by 3
- 2023LARCH: Large Language Model-based Automatic Readme Creation with Heuristicscited by 2
- 2025Automated and Context-Aware Code Documentation Leveraging Advanced LLMscited by 2
- 2026Does My README File Need To Be Updated? Exploring LLM-Based README Maintenancecited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
- 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
- 2026CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation
- 2025RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
- 2026Is Agentic RAG worth it? An experimental comparison of RAG approaches
- 2026Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
- 2024LLM Agents Improve Semantic Code Search