# TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

arXiv: https://arxiv.org/abs/2605.18025 · HTML: https://mlsanity.com/paper/2605.18025/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 67 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 30 of 34)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Graph Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2408.08921/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://mlsanity.com/paper/2404.05225/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge](https://mlsanity.com/paper/2310.15051/) (2023)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion](https://mlsanity.com/paper/2503.11576/) (2025)
- [Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.10708/) (2025)
- [MegaMath: Pushing the Limits of Open Math Corpora](https://mlsanity.com/paper/2504.02807/) (2025)
- [Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning](https://mlsanity.com/paper/2410.14211/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis](https://mlsanity.com/paper/2502.08224/) (2025)
- [TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications](https://mlsanity.com/paper/2406.01768/) (2024)
- [A Survey of AIOps for Failure Management in the Era of Large Language Models](https://mlsanity.com/paper/2406.11213/) (2024)
- [Reasoning Language Models for Root Cause Analysis in 5G Wireless Networks](https://mlsanity.com/paper/2507.21974/) (2025)
- [Large Language Models for Wireless Networks: An Overview from the Prompt Engineering Perspective](https://mlsanity.com/paper/2411.04136/) (2024)
- [Large-Scale AI in Telecom: Charting the Roadmap for Innovation, Scalability, and Enhanced Digital Experiences](https://mlsanity.com/paper/2503.04184/) (2025)
- [Towards Practical GraphRAG: Efficient Knowledge Graph Construction and Hybrid Retrieval at Scale](https://mlsanity.com/paper/2507.03226/) (2025)
- [FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs](https://mlsanity.com/paper/2501.09957/) (2025)
- [GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation](https://mlsanity.com/paper/2505.20416/) (2025)
- [The Bridge Toward 6G: 5G-Advanced Evolution in 3GPP Release 19](https://mlsanity.com/paper/2312.15174/) (2023)
- [Validating Network Protocol Parsers with Traceable RFC Document Interpretation](https://mlsanity.com/paper/2504.18050/) (2025)
- [TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation](https://mlsanity.com/paper/2601.04202/) (2025)
- [OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models](https://mlsanity.com/paper/2310.07637/) (2023)
- [Efficient Telecom Specific LLM: TSLAM-Mini with QLoRA and Digital Twin Data](https://mlsanity.com/paper/2505.07877/) (2025)
- [TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain](https://mlsanity.com/paper/2412.15891/) (2024)

## Built on by (most cited first, 1 of 1)

- [Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving](https://mlsanity.com/paper/2606.27457/) (2026)

## Abstract (arXiv)

While Large Language Models have achieved remarkable integration in various vertical scenarios, their deployment in the telecommunications domain remains exploratory due to the lack of a standardized evaluation framework. Current telecom benchmarks primarily focus on static, foundational knowledge and isolated atomic skills, neglecting the equipment-specific documentation and end-to-end industrial workflows essential for real-world production systems. To bridge this gap, we present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates LLMs across a synergistic hierarchy: (1) Multi-dimensional Knowledge Comprehension, which integrates telecommunication fundamentals, 3GPP protocols, and 5G network architecture with proprietary product knowledge across wired, core, and wireless networks via knowledge graph-driven synthesis; and (2)End-to-End Knowledge Application, which formalizes six core tasks on authentic trajectories from live network agent workflows, including intent recognition, entity extraction, event verification, tool invocation, root cause analysis, and solution generation-across network optimization and fault maintenance scenarios. Evaluations of eight state-of-the-art LLMs reveal a universal Execution Wall: while models achieve 90% accuracy in linguistic interface tasks such as intent recognition and entity extraction, performance collapses to approximately 30% in procedural execution tasks like solution generation. This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers. TeleCom-Bench provides standardized diagnostics to precisely pinpoint this deficit, offering actionable guidance for domain-specific alignment toward production-ready telecom agents. The dataset and evaluation code have been released at https://github.com/ZTE-AICloud/TeleCom-Bench.