# Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports

arXiv: https://arxiv.org/abs/2606.18166 · HTML: https://mlsanity.com/paper/2606.18166/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [SecureBERT: A Domain-Specific Language Model for Cybersecurity](https://mlsanity.com/paper/2204.02685/) (2022)
- [Evaluation Measures for Hierarchical Classification: a unified view and novel approaches](https://mlsanity.com/paper/1306.6802/) (2013)
- [AttacKG: Constructing Technique Knowledge Graph from Cyber Threat Intelligence Reports](https://mlsanity.com/paper/2111.07093/) (2021)
- [LLMCloudHunter: Harnessing LLMs for Automated Extraction of Detection Rules from Cloud-Based CTI](https://mlsanity.com/paper/2407.05194/) (2024)
- [Automated Retrieval of ATT&CK Tactics and Techniques for Cyber Threat Reports](https://mlsanity.com/paper/2004.14322/) (2020)
- [What are the attackers doing now? Automating cyber threat intelligence extraction from text on pace with the changing threat landscape: A survey](https://mlsanity.com/paper/2109.06808/) (2021)
- [CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification](https://mlsanity.com/paper/2109.04853/) (2021)
- [Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation](https://mlsanity.com/paper/2401.00280/) (2023)
- [TTPXHunter: Actionable Threat Intelligence Extraction as TTPs from Finished Cyber Threat Reports](https://mlsanity.com/paper/2403.03267/) (2024)
- [Large Language Models Are Unreliable for Cyber Threat Intelligence](https://mlsanity.com/paper/2503.23175/) (2025)
- [CTI-HAL: A Human-Annotated Dataset for Cyber Threat Intelligence Analysis](https://mlsanity.com/paper/2504.05866/) (2025)
- [Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models](https://mlsanity.com/paper/2505.03147/) (2025)
- [From Threat Reports to Continuous Threat Intelligence: A Comparison of Attack Technique Extraction Methods from Textual Artifacts](https://mlsanity.com/paper/2210.02601/) (2022)
- [Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs](https://mlsanity.com/paper/2502.02337/) (2025)
- [Mining Temporal Attack Patterns from Cyberthreat Intelligence Reports](https://mlsanity.com/paper/2401.01883/) (2024)
- [What Are Adversaries Doing? Automating Tactics, Techniques, and Procedures Extraction: A Systematic Review](https://mlsanity.com/paper/2604.02377/) (2026)
- [Beyond Single Reports: Evaluating Automated ATT&CK Technique Extraction in Multi-Report Campaign Settings](https://mlsanity.com/paper/2604.07470/) (2026)

## Built on by (most cited first, 1 of 1)

- [Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports](https://mlsanity.com/paper/2609.01671/) (2026)

## Abstract (arXiv)

Classifying Cyber Threat Intelligence (CTI) using MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is essential for proactive defense, but historically required extensive human effort. Pre-Large Language Model (LLM) automation sped up this process, but could not resolve the complex language and multi-step attack patterns found in unstructured CTI reports. LLMs addressed previous limitations by using contextual reasoning to understand unstructured text. However, current evaluations rely on simplified, single-technique sentences that ignore the complexity of real-world CTI reports, which often leads to inflated performance results. Consequently, the baseline performance of open-source LLMs on complex unstructured CTI reports remains unevaluated. To address this gap, we constructed a ground-truth dataset of 2,076 human-annotated sentences (1,281 technique-positive, 795 negative) from 83 complex unstructured CTI reports. These sentences were mapped to 114 unique ATT&CK techniques using a six-phase annotation process, achieving kappa = 0.68 inter-annotator agreement. Using this dataset, we evaluated seven open-source LLMs ranging from 8B to 236B parameters across prompt strategy and temperature configurations. The highest-performing LLM achieved a micro-averaged F1 score of 0.22, establishing the empirical baseline for multi-label ATT&CK classification on complex unstructured CTI. Parameter size showed a statistically significant positive correlation with F1 score. Prompt strategy and temperature produced no statistically significant gains across model configurations. These results indicate that current open-source LLMs are insufficient for production-grade ATT&CK classification. The dataset, benchmark, and findings provide a reproducible foundation for future CTI research.