# Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

arXiv: https://arxiv.org/abs/2504.21039 · HTML: https://mlsanity.com/paper/2504.21039/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 68 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [Synthetic continued pretraining](https://mlsanity.com/paper/2409.07431/) (2024)
- [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge](https://mlsanity.com/paper/2402.07688/) (2024)

## Built on by (most cited first, 30 of 31)

- [Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report](https://mlsanity.com/paper/2508.01059/) (2025)
- [DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experience](https://mlsanity.com/paper/2506.04282/) (2025)
- [EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora](https://mlsanity.com/paper/2506.20963/) (2025)
- [Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Report](https://mlsanity.com/paper/2601.21051/) (2026)
- [LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization](https://mlsanity.com/paper/2602.10576/) (2026)
- [On the creation of narrow AI: hierarchy and nonlocality of neural network skills](https://mlsanity.com/paper/2505.15811/) (2025)
- [MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs](https://mlsanity.com/paper/2506.22557/) (2025)
- [Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens](https://mlsanity.com/paper/2507.02964/) (2025)
- [Style Attack Disguise: When Fonts Become a Camouflage for Adversarial Intent](https://mlsanity.com/paper/2510.19641/) (2025)
- [EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering](https://mlsanity.com/paper/2511.19523/) (2025)
- [Antares: Foundation Models for Agentic Vulnerability Localization](https://mlsanity.com/paper/2608.02407/) (2026)
- [RedSage: A Cybersecurity Generalist LLM](https://mlsanity.com/paper/2601.22159/) (2026)
- [To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack](https://mlsanity.com/paper/2602.02595/) (2026)
- [A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems](https://mlsanity.com/paper/2605.17075/) (2026)
- [MOT-SR: Multi-Objective Tool-Augmented Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2607.29561/) (2026)
- [On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs](https://mlsanity.com/paper/2511.14908/) (2025)
- [DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving](https://mlsanity.com/paper/2603.00919/) (2026)
- [Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMs](https://mlsanity.com/paper/2603.14911/) (2026)
- [Probing Cultural Signals in Large Language Models through Author Profiling](https://mlsanity.com/paper/2603.16749/) (2026)
- [XekRung Technical Report](https://mlsanity.com/paper/2605.00072/) (2026)
- [LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy](https://mlsanity.com/paper/2605.04295/) (2026)
- [Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights](https://mlsanity.com/paper/2605.10808/) (2026)
- [Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR](https://mlsanity.com/paper/2606.08168/) (2026)
- [Agentra: A Supervisable Multi-Agent Framework for Enterprise Intrusion Response](https://mlsanity.com/paper/2606.18325/) (2026)
- [Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices](https://mlsanity.com/paper/2606.20571/) (2026)
- [Operationalizing Cyber Threat Intelligence with GraphRAG](https://mlsanity.com/paper/2608.13050/) (2026)
- [A Graph-Based Approach for Mapping Kernel-Level Telemetry to MITRE ATT&CK](https://mlsanity.com/paper/2609.12841/) (2026)
- [The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearning](https://mlsanity.com/paper/2609.29045/) (2026)
- [No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents](https://mlsanity.com/paper/2610.00557/) (2026)
- [Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution](https://mlsanity.com/paper/2610.00590/) (2026)

## Abstract (arXiv)

As transformer-based large language models (LLMs) increasingly permeate society, they have revolutionized domains such as software engineering, creative writing, and digital arts. However, their adoption in cybersecurity remains limited due to challenges like scarcity of specialized training data and complexity of representing cybersecurity-specific knowledge. To address these gaps, we present Foundation-Sec-8B, a cybersecurity-focused LLM built on the Llama 3.1 architecture and enhanced through continued pretraining on a carefully curated cybersecurity corpus. We evaluate Foundation-Sec-8B across both established and new cybersecurity benchmarks, showing that it matches Llama 3.1-70B and GPT-4o-mini in certain cybersecurity-specific tasks. By releasing our model to the public, we aim to accelerate progress and adoption of AI-driven tools in both public and private cybersecurity contexts.