# FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets

arXiv: https://arxiv.org/abs/2608.27021 · HTML: https://mlsanity.com/paper/2608.27021/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 8 of 8)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Can LLMs Understand Computer Networks? Towards a Virtual System Administrator](https://mlsanity.com/paper/2404.12689/) (2024)
- [NetArena: Dynamic Benchmarks for AI Agents in Network Automation](https://mlsanity.com/paper/2506.03231/) (2025)
- [A Network Arena for Benchmarking AI Agents on Network Troubleshooting](https://mlsanity.com/paper/2512.16381/) (2025)
- [TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents](https://mlsanity.com/paper/2604.06209/) (2026)
- [SADE: Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting](https://mlsanity.com/paper/2605.04530/) (2026)

## Built on by (most cited first, 1 of 1)

- [SoK: Semantic Decision Engines in Network Control Loops](https://mlsanity.com/paper/2610.06425/) (2026)

## Abstract (arXiv)

LLM-based agents are increasingly proposed for network fault diagnosis, but existing benchmarks evaluate them only on accurate tickets and always assume a fault is present, conditions rarely met in practice. We present FaulT-Bench, a benchmark of 200 troubleshooting scenarios across eight network topologies, five reimplemented from public practitioner labs, spanning genuine faults, false fault reports, incorrect device attribution, and incorrect root-cause claims. To isolate how ticket wording affects diagnosis, we further rewrite 72 false-premise tickets into five reporter personas that vary reporter confidence and verifiable detail one factor at a time, holding the network state fixed. Our automated harness deploys each scenario in Kathar'a, lets agents interact through the NIKA tool interface, and scores free-text diagnoses with an LLM judge across outcome, fix, and reasoning quality. Evaluating SADE, ReAct, and Claude Code, we find all three are near-saturated on accurate tickets and robust to misdirection, yet degrade sharply when the network is healthy and the ticket is wrong, probing until a benign condition can be promoted to a root cause rather than concluding nothing is wrong. Persona rewrites show that how a ticket is written matters more than what it claims: a confidently wrong report is handled about as well as an accurate one, while a vague, underspecified report degrades performance sharply. The three agents also fail differently, from constant over-diagnosis to unanswered runs, at very different cost. These results position FaulT-Bench as a benchmark for developing agentic systems that can reason reliably over the noisy, unreliable tickets of real-world network troubleshooting.