mlsanityData to 14 Oct 2026

TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge

arXiv 2310.15051 ↗First posted 23 Oct 2023cs.IT cs.AI cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

We introduce TeleQnA, the first benchmark dataset designed to evaluate the knowledge of Large Language Models (LLMs) in telecommunications. Comprising 10,000 questions and answers, this dataset draws from diverse sources, including standards and research articles. This paper outlines the automated question generation framework responsible for creating this dataset, along with how human input was integrated at various stages to ensure the quality of the questions. Afterwards, using the provided dataset, an evaluation is conducted to assess the capabilities of LLMs, including GPT-3.5 and GPT-4. The results highlight that these models struggle with complex standards related questions but exhibit proficiency in addressing general telecom-related inquiries. Additionally, our results showcase how incorporating telecom knowledge context significantly enhances their performance, thus shedding light on the need for a specialized telecom foundation model. Finally, the dataset is shared with active telecom professionals, whose performance is subsequently benchmarked against that of the LLMs. The findings illustrate that LLMs can rival the performance of active professionals in telecom knowledge, thanks to their capacity to process vast amounts of information, underscoring the potential of LLMs within this domain. The dataset has been made publicly accessible on GitHub.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 10 of its 22 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

10 of 22 references.

References · 10

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2023BloombergGPT: A Large Language Model for Financecited by 939
  6. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  7. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  8. 2023Large Language Models for Telecom: Forthcoming Impact on the Industrycited by 44
  9. 2023Understanding Telecom Language Through Large Language Modelscited by 32
  10. 2023Adapting Foundation Models for Information Synthesis of Wireless Communication Specificationscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 21; 2026: 16.

63 papers.

Sort:

By area:

  1. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  2. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  3. 2024WirelessLLM: Empowering Large Language Models Towards Wireless Intelligencecited by 44
  4. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 40
  5. 2024Large Multi-Modal Models (LMMs) as Universal Foundation Models for AI-Native Wireless Systemscited by 27
  6. 2024TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specificationscited by 27
  7. 2024Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunicationscited by 26
  8. 2024Tele-LLMs: A Series of Specialized Large Language Models for Telecommunicationscited by 25
  9. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  10. 2024ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networkscited by 24
  11. 2023AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systemscited by 19
  12. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
38 more
  1. 2024GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoningcited by 13
  2. 2024Using Large Language Models to Understand Telecom Standardscited by 13
  3. 2024Linguistic Intelligence in Large Language Models for Telecommunicationscited by 11
  4. 2024Telecom Language Models: Must They Be Large?cited by 10
  5. 2025Toward Agentic AI: Generative Information Retrieval Inspired Intelligent Communications and Networkingcited by 9
  6. 2024Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standardscited by 8
  7. 2024ENWAR: A RAG-empowered Multi-Modal LLM Framework for Wireless Environment Perceptioncited by 7
  8. 20266G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networkscited by 7
  9. 2026WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networkscited by 7
  10. 2025Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documentscited by 5
  11. 2025TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solvingcited by 5
  12. 2024Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6Gcited by 4
  13. 2025Mapping the Landscape of Generative AI in Network Monitoring and Managementcited by 4
  14. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selectioncited by 3
  15. 2024Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specificationscited by 3
  16. 2024NetOrchLLM: Mastering Wireless Network Orchestration with Large Language Modelscited by 3
  17. 2024TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domaincited by 3
  18. 2025Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routingcited by 3
  19. 2025WirelessMathLM: Teaching Mathematical Reasoning for LLMs in Wireless Communications with Reinforcement Learningcited by 3
  20. 2026Enhancing Large Language Models (LLMs) for Telecom using Dynamic Knowledge Graphs and Explainable Retrieval-Augmented Generationcited by 3
  21. 2024QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shufflingcited by 2
  22. 2024TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Networkcited by 2
  23. 2025Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systemscited by 2
  24. 2026TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunicationscited by 2
  25. 2025Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generationcited by 1
  26. 2025TelePlanNet: An AI-Driven Framework for Efficient Telecom Network Planningcited by 1
  27. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  28. 2025Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulationscited by 1
  29. 2025Evaluating Open-Source Large Language Models for Technical Telecom Question Answeringcited by 1
  30. 2025Reliable Inference in Edge-Cloud Model Cascades via Conformal Alignmentcited by 1
  31. 2025How to Discover Knowledge for FutureG: Contextual RAG and LLM Prompting for O-RANcited by 1
  32. 2026TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?cited by 1
  33. 2026TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Taskscited by 1
  34. 2024ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answeringcited by 0
  35. 2024WirelessAgent: Large Language Model Agents for Intelligent Wireless Networkscited by 0
  36. 2024Rephrase and Contrast: Fine-Tuning Language Models for Enhanced Understanding of Communication and Computer Networkscited by 0
  37. 2025Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Frameworkcited by 0
  38. 2025SafeCOMM: A Study on Safety Degradation in Fine-Tuned Telecom Large Language Modelscited by 0

50 most cited of 63 All 63 (JSON)