# CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain

arXiv: https://arxiv.org/abs/2212.02974 · HTML: https://mlsanity.com/paper/2212.02974/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [Portuguese Named Entity Recognition using BERT-CRF](https://mlsanity.com/paper/1909.10649/) (2019)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection](https://mlsanity.com/paper/2103.03806/) (2021)
- [CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails](https://mlsanity.com/paper/2010.03484/) (2020)
- [Continual Domain Adaptation for Machine Reading Comprehension](https://mlsanity.com/paper/2008.10874/) (2020)
- [Continual Domain-Tuning for Pretrained Language Models](https://mlsanity.com/paper/2004.02288/) (2020)
- [Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence](https://mlsanity.com/paper/2207.11076/) (2022)

## Built on by (most cited first, 30 of 31)

- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report](https://mlsanity.com/paper/2508.01059/) (2025)
- [LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledge](https://mlsanity.com/paper/2401.10036/) (2024)
- [MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context](https://mlsanity.com/paper/2407.02730/) (2024)
- [SECURE: Benchmarking Large Language Models for Cybersecurity](https://mlsanity.com/paper/2405.20441/) (2024)
- [CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions](https://mlsanity.com/paper/2408.09304/) (2024)
- [ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models](https://mlsanity.com/paper/2412.04756/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation](https://mlsanity.com/paper/2401.00280/) (2023)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English](https://mlsanity.com/paper/2212.10011/) (2022)
- [DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification](https://mlsanity.com/paper/2409.09143/) (2024)
- [Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity](https://mlsanity.com/paper/2405.16579/) (2024)
- [Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens](https://mlsanity.com/paper/2507.02964/) (2025)
- [Retrieval-Augmented LLMs for Security Incident Analysis](https://mlsanity.com/paper/2603.18196/) (2026)
- [CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuning](https://mlsanity.com/paper/2503.09334/) (2025)
- [MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Models](https://mlsanity.com/paper/2512.21231/) (2025)
- [VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use](https://mlsanity.com/paper/2605.13989/) (2026)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [Towards AI-Driven Human-Machine Co-Teaming for Adaptive and Agile Cyber Security Operation Centers](https://mlsanity.com/paper/2505.06394/) (2025)
- [SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence](https://mlsanity.com/paper/2510.00240/) (2025)
- [Alert-ME: An Explainability-Driven Defense Against Adversarial Examples in Transformer-Based Text Classification](https://mlsanity.com/paper/2307.01225/) (2023)
- [ThreatCrawl: A BERT-based Focused Crawler for the Cybersecurity Domain](https://mlsanity.com/paper/2304.11960/) (2023)
- [Innovative Research on IoT Architecture and Robotic Operating Platforms: Applications of Large Language Models and Generative AI](https://mlsanity.com/paper/2506.22477/) (2025)
- [Enhancing Android Malware Detection with Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.22750/) (2025)
- [CyGATE: Game-Theoretic Cyber Attack-Defense Engine for Patch Strategy Optimization](https://mlsanity.com/paper/2508.00478/) (2025)
- [BEACON: Behavioral Malware Classification with Large Language Model Embeddings and Deep Learning](https://mlsanity.com/paper/2509.14519/) (2025)
- [CTI Dataset Construction from Telegram](https://mlsanity.com/paper/2509.20943/) (2025)
- [IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Response](https://mlsanity.com/paper/2511.19644/) (2025)
- [CVE-TTP KG: Knowledge Graph Linking Software Vulnerabilities to Attack Behaviors](https://mlsanity.com/paper/2606.31557/) (2026)

## Abstract (arXiv)

The field of cybersecurity is evolving fast. Experts need to be informed about past, current and - in the best case - upcoming threats, because attacks are becoming more advanced, targets bigger and systems more complex. As this cannot be addressed manually, cybersecurity experts need to rely on machine learning techniques. In the texutual domain, pre-trained language models like BERT have shown to be helpful, by providing a good baseline for further fine-tuning. However, due to the domain-knowledge and many technical terms in cybersecurity general language models might miss the gist of textual information, hence doing more harm than good. For this reason, we create a high-quality dataset and present a language model specifically tailored to the cybersecurity domain, which can serve as a basic building block for cybersecurity systems that deal with natural language. The model is compared with other models based on 15 different domain-dependent extrinsic and intrinsic tasks as well as general tasks from the SuperGLUE benchmark. On the one hand, the results of the intrinsic tasks show that our model improves the internal representation space of words compared to the other models. On the other hand, the extrinsic, domain-dependent tasks, consisting of sequence tagging and classification, show that the model is best in specific application scenarios, in contrast to the others. Furthermore, we show that our approach against catastrophic forgetting works, as the model is able to retrieve the previously trained domain-independent knowledge. The used dataset and trained model are made publicly available