# Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering

arXiv: https://arxiv.org/abs/2505.04251 · HTML: https://mlsanity.com/paper/2505.04251/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 5 of its 29 references that are in the map.
- First posted on arXiv 7 May 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 5 of 5)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [LLM-based agents for automating the enhancement of user story quality: An early report](https://mlsanity.com/paper/2403.09442/) (2024)
- [RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems](https://mlsanity.com/paper/2306.01774/) (2023)

## Built on by (most cited first, 7 of 7)

- [Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines](https://mlsanity.com/paper/2510.07614/) (2025)
- [Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub](https://mlsanity.com/paper/2601.00477/) (2026)
- [When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2605.23414/) (2026)
- [A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AI](https://mlsanity.com/paper/2510.26275/) (2025)
- [Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Review](https://mlsanity.com/paper/2604.13103/) (2026)
- [Code Broker: A Multi-Agent System for Automated Code Quality Assessment](https://mlsanity.com/paper/2604.23088/) (2026)
- [Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI](https://mlsanity.com/paper/2607.17225/) (2026)

## Abstract (arXiv)

Multi-agent autonomous systems (MAS) are better at addressing challenges that spans across multiple domains than singular autonomous agents. This holds true within the field of software engineering (SE) as well. The state-of-the-art research on MAS within SE focuses on integrating LLMs at the core of autonomous agents to create LLM-based multi-agent autonomous (LMA) systems. However, the introduction of LMA systems into SE brings a plethora of challenges. One of the major challenges is the strategic allocation of tasks between humans and the LMA system in a trustworthy manner. To address this challenge, a RACI-based framework is proposed in this work in progress article, along with implementation guidelines and an example implementation of the framework. The proposed framework can facilitate efficient collaboration, ensure accountability, and mitigate potential risks associated with LLM-driven automation while aligning with the Trustworthy AI guidelines. The future steps for this work delineating the planned empirical validation method are also presented.