mlsanityData to 14 Oct 2026

Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training

arXiv 2502.11191 ↗First posted 16 Feb 2025cs.CR cs.AI cs.CLcited by 23

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable advancements in specialized fields such as finance, law, and medicine. However, in cybersecurity, we have noticed a lack of open-source datasets, with a particular lack of high-quality cybersecurity pretraining corpora, even though much research indicates that LLMs acquire their knowledge during pretraining. To address this, we present a comprehensive suite of datasets covering all major training stages, including pretraining, instruction fine-tuning, and reasoning distillation with cybersecurity-specific self-reflection data. Extensive ablation studies demonstrate their effectiveness on public cybersecurity benchmarks. In particular, continual pre-training on our dataset yields a 15.9% improvement in the aggregate score, while reasoning distillation leads to a 15.8% gain in security certification (CISSP). We will release all datasets and trained cybersecurity LLMs under the ODC-BY and MIT licenses to encourage further research in the community. For access to all datasets and model weights, please refer to https://huggingface.co/collections/trendmicro-ailab/primus-67b1fd27052b802b4af9d243.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 35 of its 58 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

35 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?25 Nov

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2025s1: Simple test-time scalingcited by 1,341
22 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  9. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  10. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  11. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  12. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  13. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  14. 2024Continual Learning for Large Language Models: A Surveycited by 147
  15. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  16. 2023Large Language Models in Law: A Surveycited by 128
  17. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  18. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  19. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  20. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  21. 2024CLAP: Learning Transferable Binary Code Representations with Natural Language Supervisioncited by 17
  22. 2022DexBERT: Effective, Task-Agnostic and Fine-grained Representation Learning of Android Bytecodecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 31
  2. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 23
  3. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  4. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Textcited by 10
  5. 2026Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Reportcited by 7
  6. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 4
  7. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  8. 2025Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement Learningcited by 4
  9. 2025Toward Cybersecurity-Expert Small Language Modelscited by 4
  10. 2025ELTEX: A Framework for Domain-Driven Synthetic Data Generationcited by 2
  11. 2025Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raterscited by 2
  12. 2026Antares: Foundation Models for Agentic Vulnerability Localizationcited by 2
11 more
  1. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
  2. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  3. 2026GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Constructioncited by 1
  4. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  5. 2026Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarkscited by 1
  6. 2025How Good LLM-Generated Password Policies Are?cited by 0
  7. 2026Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMscited by 0
  8. 2026XekRung Technical Reportcited by 0
  9. 2026Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Makingcited by 0
  10. 2026CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wildcited by 0
  11. 2026MiST: Mid-Training LLMs for Cybersecuritycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
  2. 2024SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
  3. 2026LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats
  4. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
  5. 2026CurricuLLM: Designing Personalized and Workforce-Aligned Cybersecurity Curricula Using Fine-Tuned LLMs
  6. 2024Ollabench: Evaluating LLMs' Reasoning for Human-centric Interdependent Cybersecurity
  7. 2026LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation
  8. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence