# NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

arXiv: https://arxiv.org/abs/2604.09678 · HTML: https://mlsanity.com/paper/2604.09678/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [Where LLM Agents Fail and How They can Learn From Failures](https://mlsanity.com/paper/2509.25370/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [NetLLM: Adapting Large Language Models for Networking](https://mlsanity.com/paper/2402.02338/) (2024)
- [Large Language Models for Networking: Applications, Enabling Techniques, and Challenges](https://mlsanity.com/paper/2311.17474/) (2023)
- [Why (and How) Networks Should Run Themselves](https://mlsanity.com/paper/1710.11583/) (2017)
- [Making Network Configuration Human Friendly](https://mlsanity.com/paper/2309.06342/) (2023)
- [Can LLMs Understand Computer Networks? Towards a Virtual System Administrator](https://mlsanity.com/paper/2404.12689/) (2024)
- [Large Language Models for Zero Touch Network Configuration Management](https://mlsanity.com/paper/2408.13298/) (2024)
- [CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting](https://mlsanity.com/paper/2505.18102/) (2025)
- [An LLM-based Agentic Framework for Accessible Network Control](https://mlsanity.com/paper/2509.20600/) (2025)

## Built on by (most cited first, 5 of 5)

- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations](https://mlsanity.com/paper/2608.12002/) (2026)
- [Can AI Agents Detect and Repair Artifact Drift in Network Experiments?](https://mlsanity.com/paper/2609.09849/) (2026)
- [Can AI Agents Deliver Verifiable Network-Wide Outcomes Across Authority Boundaries?](https://mlsanity.com/paper/2609.10181/) (2026)
- [SoK: Semantic Decision Engines in Network Control Loops](https://mlsanity.com/paper/2610.06425/) (2026)

## Abstract (arXiv)

As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.