mlsanityData to 14 Oct 2026

CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain

arXiv 2212.02974 ↗First posted 6 Dec 2022cs.CR cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

The field of cybersecurity is evolving fast. Experts need to be informed about past, current and - in the best case - upcoming threats, because attacks are becoming more advanced, targets bigger and systems more complex. As this cannot be addressed manually, cybersecurity experts need to rely on machine learning techniques. In the texutual domain, pre-trained language models like BERT have shown to be helpful, by providing a good baseline for further fine-tuning. However, due to the domain-knowledge and many technical terms in cybersecurity general language models might miss the gist of textual information, hence doing more harm than good. For this reason, we create a high-quality dataset and present a language model specifically tailored to the cybersecurity domain, which can serve as a basic building block for cybersecurity systems that deal with natural language. The model is compared with other models based on 15 different domain-dependent extrinsic and intrinsic tasks as well as general tasks from the SuperGLUE benchmark. On the one hand, the results of the intrinsic tasks show that our model improves the internal representation space of words compared to the other models. On the other hand, the extrinsic, domain-dependent tasks, consisting of sequence tagging and classification, show that the model is best in specific application scenarios, in contrast to the others. Furthermore, we show that our approach against catastrophic forgetting works, as the model is able to retrieve the previously trained domain-independent knowledge. The used dataset and trained model are made publicly available

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 26 of its 56 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

26 of 56 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
14 more
  1. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  2. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  3. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  4. 2016FastText.zip: Compressing text classification modelscited by 335
  5. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  6. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  7. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 73
  8. 2019Portuguese Named Entity Recognition using BERT-CRFcited by 73
  9. 2022A Domain-adaptive Pre-training Approach for Language Bias Detection in Newscited by 16
  10. 2021MalBERT: Using Transformers for Cybersecurity and Malicious Software Detectioncited by 12
  11. 2020CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emailscited by 7
  12. 2020Continual Domain Adaptation for Machine Reading Comprehensioncited by 5
  13. 2020Continual Domain-Tuning for Pretrained Language Modelscited by 4
  14. 2022Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 10; 2025: 13; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  2. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 23
  3. 2024LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledgecited by 22
  4. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Contextcited by 20
  5. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 14
  6. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  7. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  8. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  9. 2023Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generationcited by 8
  10. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  11. 2022PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in Englishcited by 5
  12. 2024DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classificationcited by 5
19 more
  1. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  2. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  3. 2026Retrieval-Augmented LLMs for Security Incident Analysiscited by 4
  4. 2025CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuningcited by 3
  5. 2025MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Modelscited by 3
  6. 2026VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Usecited by 3
  7. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  8. 2025Towards AI-Driven Human-Machine Co-Teaming for Adaptive and Agile Cyber Security Operation Centerscited by 2
  9. 2025SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligencecited by 2
  10. 2023Alert-ME: An Explainability-Driven Defense Against Adversarial Examples in Transformer-Based Text Classificationcited by 1
  11. 2023ThreatCrawl: A BERT-based Focused Crawler for the Cybersecurity Domaincited by 0
  12. 2025Innovative Research on IoT Architecture and Robotic Operating Platforms: Applications of Large Language Models and Generative AIcited by 0
  13. 2025Enhancing Android Malware Detection with Retrieval-Augmented Generationcited by 0
  14. 2025CyGATE: Game-Theoretic Cyber Attack-Defense Engine for Patch Strategy Optimizationcited by 0
  15. 2025BEACON: Behavioral Malware Classification with Large Language Model Embeddings and Deep Learningcited by 0
  16. 2025CTI Dataset Construction from Telegramcited by 0
  17. 2025IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Responsecited by 0
  18. 2026CVE-TTP KG: Knowledge Graph Linking Software Vulnerabilities to Attack Behaviorscited by 0
  19. 2026Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errorscited by 0