mlsanityData to 7 Oct 2026

ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks

arXiv 2407.06245 ↗First posted 8 Jul 2024cs.NI cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) can revolutionize how we deploy and operate Open Radio Access Networks (O-RAN) by enhancing network analytics, anomaly detection, and code generation and significantly increasing the efficiency and reliability of a plethora of O-RAN tasks. In this paper, we present ORAN-Bench-13K, the first comprehensive benchmark designed to evaluate the performance of Large Language Models (LLMs) within the context of O-RAN. Our benchmark consists of 13,952 meticulously curated multiple-choice questions generated from 116 O-RAN specification documents. We leverage a novel three-stage LLM framework, and the questions are categorized into three distinct difficulties to cover a wide spectrum of ORAN-related knowledge. We thoroughly evaluate the performance of several state-of-the-art LLMs, including Gemini, Chat-GPT, and Mistral. Additionally, we propose ORANSight, a Retrieval-Augmented Generation (RAG)-based pipeline that demonstrates superior performance on ORAN-Bench-13K compared to other tested closed-source models. Our findings indicate that current popular LLM models are not proficient in O-RAN, highlighting the need for specialized models. We observed a noticeable performance improvement when incorporating the RAG-based ORANSight pipeline, with a Macro Accuracy of 0.784 and a Weighted Accuracy of 0.776, which was on average 21.55% and 22.59% better than the other tested LLMs.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

18 of 37 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Mistral 7Bcited by 3,065
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  7. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  8. 2023TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledgecited by 62
  9. 2021Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoffcited by 30
  10. 2024TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specificationscited by 26
  11. 2023The Power of Large Language Models for Wireless Communication System Development: A Case Study on FPGA Platformscited by 20
  12. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 17
6 more
  1. 2024Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalabilitycited by 15
  2. 2023A Survey on Model-based, Heuristic, and Machine Learning Optimization Approaches in RIS-aided Wireless Networkscited by 14
  3. 2024LLM-Based Intent Processing and Network Optimization Using Attention-Based Hierarchical Reinforcement Learningcited by 13
  4. 2023Evaluating LLMs on Document-Based QA: Exact Answer Selection and Numerical Extraction using Cogtale datasetcited by 11
  5. 2023CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generationcited by 8
  6. 2024Towards Energy Efficient RAN: From Industry Standards to Trending Practicecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluationcited by 27
  2. 2025ORANSight-2.0: Foundational LLMs for O-RANcited by 13
  3. 2024Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standardscited by 8
  4. 2025ORAN-GUIDE: RAG-Driven Prompt Learning for LLM-Augmented Reinforcement Learning in O-RAN Network Slicingcited by 3
  5. 2026Enhancing Large Language Models (LLMs) for Telecom using Dynamic Knowledge Graphs and Explainable Retrieval-Augmented Generationcited by 3
  6. 2024TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Networkcited by 2
  7. 2025Prompt-Tuned LLM-Augmented DRL for Dynamic O-RAN Network Slicingcited by 2
  8. 2025Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Taskscited by 2
  9. 2025Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)cited by 2
  10. 2025Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systemscited by 2
  11. 2026TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunicationscited by 2
  12. 2025Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generationcited by 1
10 more
  1. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  2. 2025How to Discover Knowledge for FutureG: Contextual RAG and LLM Prompting for O-RANcited by 1
  3. 2026Enhancing Confidence Estimation in Telco LLMs via Twin-Pass CoT-Ensemblingcited by 0
  4. 2026TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunicationscited by 0
  5. 2026TelcoAgent: A Scalable 5G Multi-KPM Forecasting With 3GPP-Grounded Explainabilitycited by 0
  6. 2026CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operationscited by 0
  7. 2026Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judgecited by 0
  8. 2026TelecomGPT-R1: A Unified Open-Source Reasoner for the Telecom Stackcited by 0
  9. 2026CRAFT: Fine-Tuning Pre-hoc Explainability in AI-native 6G RANcited by 0
  10. 2026TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sentinels of the Stream: Unleashing Large Language Models for Dynamic Packet Classification in Software Defined Networks -- Position Paper
  2. 2025Evaluating Open-Source Large Language Models for Technical Telecom Question Answering
  3. 2025WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications
  4. 2025MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications
  5. 2024WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence
  6. 2024Telecom Language Models: Must They Be Large?
  7. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs
  8. 2025RAG-Empowered LLM-Driven Dynamic Radio Resource Management in Open 6G RAN