# TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge

arXiv: https://arxiv.org/abs/2310.15051 · HTML: https://mlsanity.com/paper/2310.15051/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [Large Language Models for Telecom: Forthcoming Impact on the Industry](https://mlsanity.com/paper/2308.06013/) (2023)
- [Understanding Telecom Language Through Large Language Models](https://mlsanity.com/paper/2306.07933/) (2023)
- [Adapting Foundation Models for Information Synthesis of Wireless Communication Specifications](https://mlsanity.com/paper/2308.04033/) (2023)

## Built on by (most cited first, 30 of 63)

- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence](https://mlsanity.com/paper/2405.17053/) (2024)
- [TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models](https://mlsanity.com/paper/2407.09424/) (2024)
- [Large Multi-Modal Models (LMMs) as Universal Foundation Models for AI-Native Wireless Systems](https://mlsanity.com/paper/2402.01748/) (2024)
- [TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications](https://mlsanity.com/paper/2406.01768/) (2024)
- [Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunications](https://mlsanity.com/paper/2404.15939/) (2024)
- [Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications](https://mlsanity.com/paper/2409.05314/) (2024)
- [SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging](https://mlsanity.com/paper/2503.17239/) (2025)
- [ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks](https://mlsanity.com/paper/2407.06245/) (2024)
- [AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systems](https://mlsanity.com/paper/2311.05842/) (2023)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoning](https://mlsanity.com/paper/2402.16631/) (2024)
- [Using Large Language Models to Understand Telecom Standards](https://mlsanity.com/paper/2404.02929/) (2024)
- [Linguistic Intelligence in Large Language Models for Telecommunications](https://mlsanity.com/paper/2402.15818/) (2024)
- [Telecom Language Models: Must They Be Large?](https://mlsanity.com/paper/2403.04666/) (2024)
- [Toward Agentic AI: Generative Information Retrieval Inspired Intelligent Communications and Networking](https://mlsanity.com/paper/2502.16866/) (2025)
- [Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards](https://mlsanity.com/paper/2408.11775/) (2024)
- [ENWAR: A RAG-empowered Multi-Modal LLM Framework for Wireless Environment Perception](https://mlsanity.com/paper/2410.18104/) (2024)
- [6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks](https://mlsanity.com/paper/2602.08675/) (2026)
- [WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networks](https://mlsanity.com/paper/2603.00501/) (2026)
- [Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documents](https://mlsanity.com/paper/2501.13954/) (2025)
- [TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving](https://mlsanity.com/paper/2506.10674/) (2025)
- [Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6G](https://mlsanity.com/paper/2404.01713/) (2024)
- [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025)
- [Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selection](https://mlsanity.com/paper/2406.00023/) (2024)
- [Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications](https://mlsanity.com/paper/2410.19790/) (2024)
- [NetOrchLLM: Mastering Wireless Network Orchestration with Large Language Models](https://mlsanity.com/paper/2412.10107/) (2024)
- [TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain](https://mlsanity.com/paper/2412.15891/) (2024)
- [Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing](https://mlsanity.com/paper/2505.11856/) (2025)

## Abstract (arXiv)

We introduce TeleQnA, the first benchmark dataset designed to evaluate the knowledge of Large Language Models (LLMs) in telecommunications. Comprising 10,000 questions and answers, this dataset draws from diverse sources, including standards and research articles. This paper outlines the automated question generation framework responsible for creating this dataset, along with how human input was integrated at various stages to ensure the quality of the questions. Afterwards, using the provided dataset, an evaluation is conducted to assess the capabilities of LLMs, including GPT-3.5 and GPT-4. The results highlight that these models struggle with complex standards related questions but exhibit proficiency in addressing general telecom-related inquiries. Additionally, our results showcase how incorporating telecom knowledge context significantly enhances their performance, thus shedding light on the need for a specialized telecom foundation model. Finally, the dataset is shared with active telecom professionals, whose performance is subsequently benchmarked against that of the LLMs. The findings illustrate that LLMs can rival the performance of active professionals in telecom knowledge, thanks to their capacity to process vast amounts of information, underscoring the potential of LLMs within this domain. The dataset has been made publicly accessible on GitHub.