# An Empirical Study of NetOps Capability of Pre-Trained Large Language Models

arXiv: https://arxiv.org/abs/2309.05557 · HTML: https://mlsanity.com/paper/2309.05557/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)

## Built on by (most cited first, 9 of 9)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence](https://mlsanity.com/paper/2405.17053/) (2024)
- [SECURE: Benchmarking Large Language Models for Cybersecurity](https://mlsanity.com/paper/2405.20441/) (2024)
- [Linguistic Intelligence in Large Language Models for Telecommunications](https://mlsanity.com/paper/2402.15818/) (2024)
- [GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configuration](https://mlsanity.com/paper/2407.08249/) (2024)
- [EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents](https://mlsanity.com/paper/2601.17722/) (2026)
- [OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models](https://mlsanity.com/paper/2310.07637/) (2023)
- [AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2511.01144/) (2025)
- [InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing](https://mlsanity.com/paper/2505.22156/) (2025)

## Abstract (arXiv)

Nowadays, the versatile capabilities of Pre-trained Large Language Models (LLMs) have attracted much attention from the industry. However, some vertical domains are more interested in the in-domain capabilities of LLMs. For the Networks domain, we present NetEval, an evaluation set for measuring the comprehensive capabilities of LLMs in Network Operations (NetOps). NetEval is designed for evaluating the commonsense knowledge and inference ability in NetOps in a multi-lingual context. NetEval consists of 5,732 questions about NetOps, covering five different sub-domains of NetOps. With NetEval, we systematically evaluate the NetOps capability of 26 publicly available LLMs. The results show that only GPT-4 can achieve a performance competitive to humans. However, some open models like LLaMA 2 demonstrate significant potential.