# Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering

arXiv: https://arxiv.org/abs/2608.24271 · HTML: https://mlsanity.com/paper/2608.24271/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 9 of 9)

- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?](https://mlsanity.com/paper/2509.03312/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale](https://mlsanity.com/paper/2409.16299/) (2024)
- [AgentOps: Enabling Observability of LLM Agents](https://mlsanity.com/paper/2411.05285/) (2024)
- [Aegis: Automated Error Generation and Attribution for Multi-Agent Systems](https://mlsanity.com/paper/2509.14295/) (2025)
- [LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Systems](https://mlsanity.com/paper/2508.12412/) (2025)

## Built on by (most cited first, 3 of 3)

- [Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Fault](https://mlsanity.com/paper/2609.32802/) (2026)
- [Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents](https://mlsanity.com/paper/2610.04375/) (2026)
- [When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2610.15459/) (2026)

## Abstract (arXiv)

Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-agent systems in software engineering workflows. The tool instruments agent executions with trace-aligned telemetry across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-agent system for software development.