# SecureBERT: A Domain-Specific Language Model for Cybersecurity

arXiv: https://arxiv.org/abs/2204.02685 · HTML: https://mlsanity.com/paper/2204.02685/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Adversarial Noise Layer: Regularize Neural Network By Adding Noise](https://mlsanity.com/paper/1805.08000/) (2018)
- [V2W-BERT: A Framework for Effective Hierarchical Multiclass Classification of Software Vulnerabilities](https://mlsanity.com/paper/2102.11498/) (2021)
- [Host-based anomaly detection using Eigentraces feature extraction and one-class classification on system call trace data](https://mlsanity.com/paper/1911.11284/) (2019)

## Built on by (most cited first, 30 of 60)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices](https://mlsanity.com/paper/2306.14263/) (2023)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training](https://mlsanity.com/paper/2502.11191/) (2025)
- [Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild](https://mlsanity.com/paper/2307.10214/) (2023)
- [SECURE: Benchmarking Large Language Models for Cybersecurity](https://mlsanity.com/paper/2405.20441/) (2024)
- [Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2407.02528/) (2024)
- [CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2410.21060/) (2024)
- [CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions](https://mlsanity.com/paper/2408.09304/) (2024)
- [On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions](https://mlsanity.com/paper/2306.14062/) (2023)
- [ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models](https://mlsanity.com/paper/2412.04756/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation](https://mlsanity.com/paper/2401.00280/) (2023)
- [MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation](https://mlsanity.com/paper/2407.15748/) (2024)
- [MoE2: Optimizing Collaborative Inference for Edge Large Language Models](https://mlsanity.com/paper/2501.09410/) (2025)
- [DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification](https://mlsanity.com/paper/2409.09143/) (2024)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [CVE-driven Attack Technique Prediction with Semantic Information Extraction and a Domain-specific Language Model](https://mlsanity.com/paper/2309.02785/) (2023)
- [Automated CVE Analysis for Threat Prioritization and Impact Prediction](https://mlsanity.com/paper/2309.03040/) (2023)
- [CANAL -- Cyber Activity News Alerting Language Model: Empirical Approach vs. Expensive LLM](https://mlsanity.com/paper/2405.06772/) (2024)
- [BARTPredict: Empowering IoT Security with LLM-Driven Cyber Threat Prediction](https://mlsanity.com/paper/2501.01664/) (2025)
- [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://mlsanity.com/paper/2503.04302/) (2025)
- [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025)
- [Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens](https://mlsanity.com/paper/2507.02964/) (2025)
- [Retrieval-Augmented LLMs for Security Incident Analysis](https://mlsanity.com/paper/2603.18196/) (2026)
- [SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classification](https://mlsanity.com/paper/2311.12074/) (2023)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)

## Abstract (arXiv)

Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the world's economic and cultural infrastructures, while they have introduced risks in terms of cyber attacks. CTI is information that helps cybersecurity analysts make intelligent security decisions, that is often delivered in the form of natural language text, which must be transformed to machine readable format through an automated procedure before it can be used for automated security measures. This paper proposes SecureBERT, a cybersecurity language model capable of capturing text connotations in cybersecurity text (e.g., CTI) and therefore successful in automation for many critical cybersecurity tasks that would otherwise rely on human expertise and time-consuming manual efforts. SecureBERT has been trained using a large corpus of cybersecurity text.To make SecureBERT effective not just in retaining general English understanding, but also when applied to text with cybersecurity implications, we developed a customized tokenizer as well as a method to alter pre-trained weights. The SecureBERT is evaluated using the standard Masked Language Model (MLM) test as well as two additional standard NLP tasks. Our evaluation studies show that SecureBERTfootnoteurlhttps://github.com/ehsanaghaei/SecureBERT outperforms existing similar models, confirming its capability for solving crucial NLP tasks in cybersecurity.