# ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks

arXiv: https://arxiv.org/abs/2407.06245 · HTML: https://mlsanity.com/paper/2407.06245/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge](https://mlsanity.com/paper/2310.15051/) (2023)
- [Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/2102.09527/) (2021)
- [TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications](https://mlsanity.com/paper/2406.01768/) (2024)
- [The Power of Large Language Models for Wireless Communication System Development: A Case Study on FPGA Platforms](https://mlsanity.com/paper/2307.07319/) (2023)
- [Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions](https://mlsanity.com/paper/2404.09135/) (2024)
- [A Survey on Model-based, Heuristic, and Machine Learning Optimization Approaches in RIS-aided Wireless Networks](https://mlsanity.com/paper/2303.14320/) (2023)
- [Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability](https://mlsanity.com/paper/2406.11424/) (2024)
- [LLM-Based Intent Processing and Network Optimization Using Attention-Based Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2406.06059/) (2024)
- [Evaluating LLMs on Document-Based QA: Exact Answer Selection and Numerical Extraction using Cogtale dataset](https://mlsanity.com/paper/2311.07878/) (2023)
- [CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation](https://mlsanity.com/paper/2401.06786/) (2023)
- [Towards Energy Efficient RAN: From Industry Standards to Trending Practice](https://mlsanity.com/paper/2402.11993/) (2024)

## Built on by (most cited first, 24 of 24)

- [Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation](https://mlsanity.com/paper/2502.06563/) (2025)
- [ORANSight-2.0: Foundational LLMs for O-RAN](https://mlsanity.com/paper/2503.05200/) (2025)
- [Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards](https://mlsanity.com/paper/2408.11775/) (2024)
- [ORAN-GUIDE: RAG-Driven Prompt Learning for LLM-Augmented Reinforcement Learning in O-RAN Network Slicing](https://mlsanity.com/paper/2506.00576/) (2025)
- [Enhancing Large Language Models (LLMs) for Telecom using Dynamic Knowledge Graphs and Explainable Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.17529/) (2026)
- [TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network](https://mlsanity.com/paper/2411.02617/) (2024)
- [Prompt-Tuned LLM-Augmented DRL for Dynamic O-RAN Network Slicing](https://mlsanity.com/paper/2506.00574/) (2025)
- [Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks](https://mlsanity.com/paper/2506.20009/) (2025)
- [Benchmarking Vector, Graph and Hybrid Retrieval Augmented Generation (RAG) Pipelines for Open Radio Access Networks (ORAN)](https://mlsanity.com/paper/2507.03608/) (2025)
- [Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems](https://mlsanity.com/paper/2512.20012/) (2025)
- [TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunications](https://mlsanity.com/paper/2605.09929/) (2026)
- [Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.24245/) (2025)
- [Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap](https://mlsanity.com/paper/2508.18646/) (2025)
- [How to Discover Knowledge for FutureG: Contextual RAG and LLM Prompting for O-RAN](https://mlsanity.com/paper/2601.02382/) (2025)
- [TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks](https://mlsanity.com/paper/2609.25356/) (2026)
- [Enhancing Confidence Estimation in Telco LLMs via Twin-Pass CoT-Ensembling](https://mlsanity.com/paper/2604.13271/) (2026)
- [TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications](https://mlsanity.com/paper/2604.17778/) (2026)
- [TelcoAgent: A Scalable 5G Multi-KPM Forecasting With 3GPP-Grounded Explainability](https://mlsanity.com/paper/2606.19821/) (2026)
- [CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations](https://mlsanity.com/paper/2608.12002/) (2026)
- [Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge](https://mlsanity.com/paper/2608.21021/) (2026)
- [TelecomGPT-R1: A Unified Open-Source Reasoner for the Telecom Stack](https://mlsanity.com/paper/2608.26126/) (2026)
- [CRAFT: Fine-Tuning Pre-hoc Explainability in AI-native 6G RAN](https://mlsanity.com/paper/2609.00590/) (2026)
- [OTel: Open Telco AI Datasets, Benchmarks, and Models](https://mlsanity.com/paper/2610.07766/) (2026)
- [AutoSec-RAN: Specification-Grounded LLM Fine-Tuning for Automated Security Analysis of O-RAN](https://mlsanity.com/paper/2610.15995/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) can revolutionize how we deploy and operate Open Radio Access Networks (O-RAN) by enhancing network analytics, anomaly detection, and code generation and significantly increasing the efficiency and reliability of a plethora of O-RAN tasks. In this paper, we present ORAN-Bench-13K, the first comprehensive benchmark designed to evaluate the performance of Large Language Models (LLMs) within the context of O-RAN. Our benchmark consists of 13,952 meticulously curated multiple-choice questions generated from 116 O-RAN specification documents. We leverage a novel three-stage LLM framework, and the questions are categorized into three distinct difficulties to cover a wide spectrum of ORAN-related knowledge. We thoroughly evaluate the performance of several state-of-the-art LLMs, including Gemini, Chat-GPT, and Mistral. Additionally, we propose ORANSight, a Retrieval-Augmented Generation (RAG)-based pipeline that demonstrates superior performance on ORAN-Bench-13K compared to other tested closed-source models. Our findings indicate that current popular LLM models are not proficient in O-RAN, highlighting the need for specialized models. We observed a noticeable performance improvement when incorporating the RAG-based ORANSight pipeline, with a Macro Accuracy of 0.784 and a Weighted Accuracy of 0.776, which was on average 21.55% and 22.59% better than the other tested LLMs.