mlsanityData to 7 Oct 2026

Evaluating Open-Source Large Language Models for Technical Telecom Question Answering

arXiv 2509.21949 ↗First posted 26 Sep 2025cs.NI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities across various fields. However, their performance in technical domains such as telecommunications remains underexplored. This paper evaluates two open-source LLMs, Gemma 3 27B and DeepSeek R1 32B, on factual and reasoning-based questions derived from advanced wireless communications material. We construct a benchmark of 105 question-answer pairs and assess performance using lexical metrics, semantic similarity, and LLM-as-a-judge scoring. We also analyze consistency, judgment reliability, and hallucination through source attribution and score variance. Results show that Gemma excels in semantic fidelity and LLM-rated correctness, while DeepSeek demonstrates slightly higher lexical consistency. Additional findings highlight current limitations in telecom applications and the need for domain-adapted models to support trustworthy Artificial Intelligence (AI) assistants in engineering.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 10 of its 16 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

10 of 16 references.

References · 10

  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  3. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 62
  4. 2023Large Generative AI Models for Telecom: The Next Big Thing?cited by 52
  5. 2023Large Language Models for Telecom: Forthcoming Impact on the Industrycited by 44
  6. 2023Understanding Telecom Language Through Large Language Modelscited by 32
  7. 2024Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunicationscited by 26
  8. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  9. 2023Observations on LLMs for Telecom Domain: Capabilities and Limitationscited by 11
  10. 2024Linguistic Intelligence in Large Language Models for Telecommunicationscited by 11

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systemscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026H2LooP Telecom Model v1: From Telecom Comprehension to Autonomous Issue and PR Resolution
  2. 2025Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation
  3. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  4. 2025TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving
  5. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
  6. 2024QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
  7. 2025Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering
  8. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth