mlsanityData to 7 Oct 2026

Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

arXiv 2504.21039 ↗First posted 28 Apr 2025cs.CR cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

As transformer-based large language models (LLMs) increasingly permeate society, they have revolutionized domains such as software engineering, creative writing, and digital arts. However, their adoption in cybersecurity remains limited due to challenges like scarcity of specialized training data and complexity of representing cybersecurity-specific knowledge. To address these gaps, we present Foundation-Sec-8B, a cybersecurity-focused LLM built on the Llama 3.1 architecture and enhanced through continued pretraining on a carefully curated cybersecurity corpus. We evaluate Foundation-Sec-8B across both established and new cybersecurity benchmarks, showing that it matches Llama 3.1-70B and GPT-4o-mini in certain cybersecurity-specific tasks. By releasing our model to the public, we aim to accelerate progress and adoption of AI-driven tools in both public and private cybersecurity contexts.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 46 of its 68 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

46 of 68 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
34 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2023Llemma: An Open Language Model For Mathematicscited by 385
  8. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  9. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  10. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  11. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  12. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  13. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  14. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  15. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  16. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  17. 2024Synthetic continued pretrainingcited by 56
  18. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  19. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  20. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  21. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  22. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
  23. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  24. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  25. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  26. 2024SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecuritycited by 21
  27. 2019Efficient Adaptation of Pretrained Transformers for Abstractive Summarizationcited by 13
  28. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 13
  29. 2022Continued Pretraining for Better Zero- and Few-Shot Promptabilitycited by 10
  30. 2024Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapsecited by 4
  31. 2024Architectural Selection Framework for Synthetic Network Traffic: Quantifying the Fidelity-Utility Trade-offcited by 4
  32. 2025Generative Active Adaptation for Drifting and Imbalanced Network Intrusion Detectioncited by 4
  33. 2025Continual Pre-Training is (not) What You Need in Domain Adaptioncited by 2
  34. 2024Thwarting Cybersecurity Attacks with Explainable Concept Driftcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 19.

28 papers.

Sort:

By area:

  1. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  2. 2025DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experiencecited by 20
  3. 2025EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corporacited by 14
  4. 2026Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Reportcited by 7
  5. 2025On the creation of narrow AI: hierarchy and nonlocality of neural network skillscited by 4
  6. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  7. 2026LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimizationcited by 4
  8. 2025Style Attack Disguise: When Fonts Become a Camouflage for Adversarial Intentcited by 2
  9. 2025EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answeringcited by 2
  10. 2026Antares: Foundation Models for Agentic Vulnerability Localizationcited by 2
  11. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 1
  12. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
16 more
  1. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  2. 2026MOT-SR: Multi-Objective Tool-Augmented Scientific Equation Discovery with Large Language Modelscited by 1
  3. 2025On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTscited by 0
  4. 2026DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Drivingcited by 0
  5. 2026Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMscited by 0
  6. 2026Probing Cultural Signals in Large Language Models through Author Profilingcited by 0
  7. 2026XekRung Technical Reportcited by 0
  8. 2026LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropycited by 0
  9. 2026Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insightscited by 0
  10. 2026Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDRcited by 0
  11. 2026Agentra: A Supervisable Multi-Agent Framework for Enterprise Intrusion Responsecited by 0
  12. 2026Operationalizing Cyber Threat Intelligence with GraphRAGcited by 0
  13. 2026A Graph-Based Approach for Mapping Kernel-Level Telemetry to MITRE ATT&CKcited by 0
  14. 2026The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearningcited by 0
  15. 2026No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agentscited by 0
  16. 2026Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Executioncited by 0