mlsanityData to 7 Oct 2026

SecureBERT: A Domain-Specific Language Model for Cybersecurity

arXiv 2204.02685 ↗First posted 6 Apr 2022cs.CL cs.AI cs.CRcited by 60

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) has recently gained wide attention in cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber automation. Increased connection and automation have revolutionized the world's economic and cultural infrastructures, while they have introduced risks in terms of cyber attacks. CTI is information that helps cybersecurity analysts make intelligent security decisions, that is often delivered in the form of natural language text, which must be transformed to machine readable format through an automated procedure before it can be used for automated security measures. This paper proposes SecureBERT, a cybersecurity language model capable of capturing text connotations in cybersecurity text (e.g., CTI) and therefore successful in automation for many critical cybersecurity tasks that would otherwise rely on human expertise and time-consuming manual efforts. SecureBERT has been trained using a large corpus of cybersecurity text.To make SecureBERT effective not just in retaining general English understanding, but also when applied to text with cybersecurity implications, we developed a customized tokenizer as well as a method to alter pre-trained weights. The SecureBERT is evaluated using the standard Masked Language Model (MLM) test as well as two additional standard NLP tasks. Our evaluation studies show that SecureBERTfootnoteurlhttps://github.com/ehsanaghaei/SecureBERT outperforms existing similar models, confirming its capability for solving crucial NLP tasks in cybersecurity.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 15 of its 34 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

15 of 34 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2017Towards Robust Neural Networks via Random Self-ensemblecited by 158
  11. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  12. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
3 more
  1. 2018Adversarial Noise Layer: Regularize Neural Network By Adding Noisecited by 25
  2. 2021V2W-BERT: A Framework for Effective Hierarchical Multiclass Classification of Software Vulnerabilitiescited by 13
  3. 2019Host-based anomaly detection using Eigentraces feature extraction and one-class classification on system call trace datacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 16; 2026: 18.

60 papers.

Sort:

By area:

  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  2. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  3. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  4. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  5. 2023Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devicescited by 38
  6. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  7. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  8. 2023Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wildcited by 15
  9. 2024Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Modelscited by 14
  10. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 13
  11. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  12. 2024CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Modelscited by 12
38 more
  1. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  2. 2023On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptionscited by 10
  3. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  4. 2023Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generationcited by 8
  5. 2024MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generationcited by 8
  6. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Modelscited by 6
  7. 2024DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classificationcited by 5
  8. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  9. 2023CVE-driven Attack Technique Prediction with Semantic Information Extraction and a Domain-specific Language Modelcited by 4
  10. 2023Automated CVE Analysis for Threat Prioritization and Impact Predictioncited by 4
  11. 2024CANAL -- Cyber Activity News Alerting Language Model: Empirical Approach vs. Expensive LLMcited by 4
  12. 2025BARTPredict: Empowering IoT Security with LLM-Driven Cyber Threat Predictioncited by 4
  13. 2025Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluationcited by 4
  14. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  15. 2026Retrieval-Augmented LLMs for Security Incident Analysiscited by 4
  16. 2025From Texts to Shields: Convergence of Large Language Models and Cybersecuritycited by 3
  17. 2023SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classificationcited by 2
  18. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  19. 2024CPE-Identifier: Automated CPE identification and CVE summaries annotation with Deep Learning and NLPcited by 2
  20. 2025False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systemscited by 2
  21. 2025SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligencecited by 2
  22. 2023Constructing a Knowledge Graph from Textual Descriptions of Software Vulnerabilities in the National Vulnerability Databasecited by 1
  23. 2024Decoding BACnet Packets: A Large Language Model Approach for Packet Interpretationcited by 1
  24. 2025EventHunter: Dynamic Clustering and Ranking of Security Events from Hacker Forum Discussionscited by 1
  25. 2025Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?cited by 1
  26. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
  27. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  28. 2026LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detectioncited by 1
  29. 2026LanG -- A Governance-Aware Agentic AI Platform for Unified Security Operationscited by 1
  30. 2026TIJERE: A Novel Threat Intelligence Joint Extraction Model Based on Analyst Expert Knowledgecited by 1
  31. 2026Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphscited by 1
  32. 2026Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reportscited by 1
  33. 2024Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Makingcited by 0
  34. 2024Automated CVE Analysis: Harnessing Machine Learning In Designing Question-Answering Models For Cybersecurity Information Extractioncited by 0
  35. 2025On Automating Security Policies with Contemporary LLMscited by 0
  36. 2025PlantDeBERTa: An Open Source Language Model for Plant Sciencecited by 0
  37. 2025LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Codecited by 0
  38. 2025Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMscited by 0

50 most cited of 60 All 60 (JSON)