# Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices

arXiv: https://arxiv.org/abs/2306.14263 · HTML: https://mlsanity.com/paper/2306.14263/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 41 references that are in the map.
- First posted on arXiv 25 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [SecureBERT: A Domain-Specific Language Model for Cybersecurity](https://mlsanity.com/paper/2204.02685/) (2022)
- [Transformer-Based Language Models for Software Vulnerability Detection](https://mlsanity.com/paper/2204.03214/) (2022)
- [CAN-BERT do it? Controller Area Network Intrusion Detection System based on BERT Language Model](https://mlsanity.com/paper/2210.09439/) (2022)
- [Generative AI for Cyber Threat-Hunting in 6G-enabled IoT Networks](https://mlsanity.com/paper/2303.11751/) (2023)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)

## Built on by (most cited first, 30 of 39)

- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [Towards Explainable Network Intrusion Detection using Large Language Models](https://mlsanity.com/paper/2408.04342/) (2024)
- [Review of Generative AI Methods in Cybersecurity](https://mlsanity.com/paper/2403.08701/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [The Internet of Things in the Era of Generative AI: Vision and Challenges](https://mlsanity.com/paper/2401.01923/) (2024)
- [Efficient Federated Intrusion Detection in 5G ecosystem using optimized BERT-based model](https://mlsanity.com/paper/2409.19390/) (2024)
- [Beyond Detection: Leveraging Large Language Models for Cyber Attack Prediction in IoT Networks](https://mlsanity.com/paper/2408.14045/) (2024)
- [Integrating Large Language Models with Internet of Things Applications](https://mlsanity.com/paper/2410.19223/) (2024)
- [MoE2: Optimizing Collaborative Inference for Edge Large Language Models](https://mlsanity.com/paper/2501.09410/) (2025)
- [eX-NIDS: A Framework for Explainable Network Intrusion Detection Leveraging Large Language Models](https://mlsanity.com/paper/2507.16241/) (2025)
- [Toward Intelligent and Secure Cloud: Large Language Model Empowered Proactive Defense](https://mlsanity.com/paper/2412.21051/) (2024)
- [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025)
- [ChatIoT: Large Language Model-based Security Assistant for Internet of Things with Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.09896/) (2025)
- [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://mlsanity.com/paper/2503.04302/) (2025)
- [DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response](https://mlsanity.com/paper/2505.19973/) (2025)
- [Evaluating Language Models For Threat Detection in IoT Security Logs](https://mlsanity.com/paper/2507.02390/) (2025)
- [CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuning](https://mlsanity.com/paper/2503.09334/) (2025)
- [Generative Artificial Intelligence for Internet of Things Computing: A Systematic Survey](https://mlsanity.com/paper/2504.07635/) (2025)
- [Gen-AI for User Safety: A Survey](https://mlsanity.com/paper/2411.06606/) (2024)
- [Blockchain Meets LLMs: A Living Survey on Bidirectional Integration](https://mlsanity.com/paper/2411.16809/) (2024)
- [A Unified Framework for Context-Aware IoT Management and State-of-the-Art IoT Traffic Anomaly Detection](https://mlsanity.com/paper/2412.19830/) (2024)
- [Large Language Model-driven Security Assistant for Internet of Things via Chain-of-Thought](https://mlsanity.com/paper/2505.06307/) (2025)
- [Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems](https://mlsanity.com/paper/2505.08816/) (2025)
- [Interpretable Anomaly-Based DDoS Detection in AI-RAN with XAI and LLMs](https://mlsanity.com/paper/2507.21193/) (2025)
- [Generative AI for Internet of Things Security: Challenges and Opportunities](https://mlsanity.com/paper/2502.08886/) (2025)
- [Large Language Models powered Malicious Traffic Detection: Architecture, Opportunities and Case Study](https://mlsanity.com/paper/2503.18487/) (2025)
- [LanG -- A Governance-Aware Agentic AI Platform for Unified Security Operations](https://mlsanity.com/paper/2604.05440/) (2026)

## Abstract (arXiv)

The field of Natural Language Processing (NLP) is currently undergoing a revolutionary transformation driven by the power of pre-trained Large Language Models (LLMs) based on groundbreaking Transformer architectures. As the frequency and diversity of cybersecurity attacks continue to rise, the importance of incident detection has significantly increased. IoT devices are expanding rapidly, resulting in a growing need for efficient techniques to autonomously identify network-based attacks in IoT networks with both high precision and minimal computational requirements. This paper presents SecurityBERT, a novel architecture that leverages the Bidirectional Encoder Representations from Transformers (BERT) model for cyber threat detection in IoT networks. During the training of SecurityBERT, we incorporated a novel privacy-preserving encoding technique called Privacy-Preserving Fixed-Length Encoding (PPFLE). We effectively represented network traffic data in a structured format by combining PPFLE with the Byte-level Byte-Pair Encoder (BBPE) Tokenizer. Our research demonstrates that SecurityBERT outperforms traditional Machine Learning (ML) and Deep Learning (DL) methods, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), in cyber threat detection. Employing the Edge-IIoTset cybersecurity dataset, our experimental analysis shows that SecurityBERT achieved an impressive 98.2% overall accuracy in identifying fourteen distinct attack types, surpassing previous records set by hybrid solutions such as GAN-Transformer-based architectures and CNN-LSTM models. With an inference time of less than 0.15 seconds on an average CPU and a compact model size of just 16.7MB, SecurityBERT is ideally suited for real-life traffic analysis and a suitable choice for deployment on resource-constrained IoT devices.