# Can LLMs Understand Computer Networks? Towards a Virtual System Administrator

arXiv: https://arxiv.org/abs/2404.12689 · HTML: https://mlsanity.com/paper/2404.12689/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [Deep Learning in Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/1803.04311/) (2018)
- [LLM Agents can Autonomously Hack Websites](https://mlsanity.com/paper/2402.06664/) (2024)
- [GraphLLM: Boosting Graph Reasoning Ability of Large Language Model](https://mlsanity.com/paper/2310.05845/) (2023)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent](https://mlsanity.com/paper/2312.10003/) (2023)
- [Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT](https://mlsanity.com/paper/2304.11116/) (2023)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [Large Generative AI Models for Telecom: The Next Big Thing?](https://mlsanity.com/paper/2306.10249/) (2023)
- [Large Language Models for Telecom: Forthcoming Impact on the Industry](https://mlsanity.com/paper/2308.06013/) (2023)
- [Understanding Telecom Language Through Large Language Models](https://mlsanity.com/paper/2306.07933/) (2023)
- [Machine Learning for Networking: Workflow, Advances and Opportunities](https://mlsanity.com/paper/1709.08339/) (2017)
- [SoK: Explainable Machine Learning for Computer Security Applications](https://mlsanity.com/paper/2208.10605/) (2022)
- [Enhancing Network Management Using Code Generated by Large Language Models](https://mlsanity.com/paper/2308.06261/) (2023)
- [Evaluating Generative Models for Graph-to-Text Generation](https://mlsanity.com/paper/2307.14712/) (2023)
- [Linguistic Intelligence in Large Language Models for Telecommunications](https://mlsanity.com/paper/2402.15818/) (2024)
- [Wordflow: Social Prompt Engineering for Large Language Models](https://mlsanity.com/paper/2401.14447/) (2024)

## Built on by (most cited first, 11 of 11)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Towards Explainable Network Intrusion Detection using Large Language Models](https://mlsanity.com/paper/2408.04342/) (2024)
- [The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding](https://mlsanity.com/paper/2502.08946/) (2025)
- [GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configuration](https://mlsanity.com/paper/2407.08249/) (2024)
- [eX-NIDS: A Framework for Explainable Network Intrusion Detection Leveraging Large Language Models](https://mlsanity.com/paper/2507.16241/) (2025)
- [NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration](https://mlsanity.com/paper/2604.09678/) (2026)
- [SADE: Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting](https://mlsanity.com/paper/2605.04530/) (2026)
- [FastRAG: Retrieval Augmented Generation for Semi-structured Data](https://mlsanity.com/paper/2411.13773/) (2024)
- [Can LLMs Assist Computer Education? an Empirical Case Study of DeepSeek](https://mlsanity.com/paper/2504.00421/) (2025)
- [FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets](https://mlsanity.com/paper/2608.27021/) (2026)
- [An LLM-Powered AI Agent Framework for Holistic IoT Traffic Interpretation](https://mlsanity.com/paper/2510.13925/) (2025)

## Abstract (arXiv)

Recent advancements in Artificial Intelligence, and particularly Large Language Models (LLMs), offer promising prospects for aiding system administrators in managing the complexity of modern networks. However, despite this potential, a significant gap exists in the literature regarding the extent to which LLMs can understand computer networks. Without empirical evidence, system administrators might rely on these models without assurance of their efficacy in performing network-related tasks accurately. In this paper, we are the first to conduct an exhaustive study on LLMs' comprehension of computer networks. We formulate several research questions to determine whether LLMs can provide correct answers when supplied with a network topology and questions on it. To assess them, we developed a thorough framework for evaluating LLMs' capabilities in various network-related tasks. We evaluate our framework on multiple computer networks employing proprietary (e.g., GPT4) and open-source (e.g., Llama2) models. Our findings in general purpose LLMs using a zero-shot scenario demonstrate promising results, with the best model achieving an average accuracy of 79.3%. Proprietary LLMs achieve noteworthy results in small and medium networks, while challenges persist in comprehending complex network topologies, particularly for open-source models. Moreover, we provide insight into how prompt engineering can enhance the accuracy of some tasks.