# TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

arXiv: https://arxiv.org/abs/2604.06209 · HTML: https://mlsanity.com/paper/2604.06209/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agentic AI for wireless networks, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 7 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- LLM agents and tool use / [Agentic AI for wireless networks](https://mlsanity.com/agents/agentic-ai-for-wireless-networks/)

## Builds on (most cited first, 6 of 6)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis](https://mlsanity.com/paper/2402.11398/) (2024)
- [Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance](https://mlsanity.com/paper/2412.17321/) (2024)

## Built on by (most cited first, 4 of 4)

- [TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunications](https://mlsanity.com/paper/2605.09929/) (2026)
- [OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents](https://mlsanity.com/paper/2608.08775/) (2026)
- [FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets](https://mlsanity.com/paper/2608.27021/) (2026)
- [CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations](https://mlsanity.com/paper/2608.12002/) (2026)

## Abstract (arXiv)

The integration of large language model (LLM) agents into telecom networks introduces new challenges, related to intent recognition, tool execution, and resolution generation, while taking into consideration different operational constraints. In this paper, we introduce TelcoAgent-Bench and TelcoAgent-Metrics, a Telecom-specific benchmarking framework for evaluating multilingual telecom LLM agents. The proposed framework assesses the semantic understanding as well as process-level alignment with structured troubleshooting flows and stability across repeated scenario variations. Our contribution includes a structured suite of metrics that assess intent recognition, ordered tool execution, resolution correctness, and stability across scenario variations, with the aim of quantifying the reliability and operational consistency of LLM agents in telecom environments. The framework is designed to operate in both English and Arabic, to address the need for multilingual agent deployment in operational network environments. Our experimental results show that although recent instruct-tuned models can understand telecom problems in a reasonable way, they usually struggle to consistently follow the required troubleshooting steps and to maintain stable behavior when exposed to different variations of the same scenario. This performance gap becomes more pronounced in unconstrained and bilingual settings.