# Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation

arXiv: https://arxiv.org/abs/2401.00280 · HTML: https://mlsanity.com/paper/2401.00280/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 30 Dec 2023.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [SecureBERT: A Domain-Specific Language Model for Cybersecurity](https://mlsanity.com/paper/2204.02685/) (2022)
- [CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain](https://mlsanity.com/paper/2212.02974/) (2022)
- [Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Study](https://mlsanity.com/paper/2208.12144/) (2022)
- [On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions](https://mlsanity.com/paper/2306.14062/) (2023)
- [Towards Automated Classification of Attackers' TTPs by combining NLP with ML Techniques](https://mlsanity.com/paper/2207.08478/) (2022)

## Built on by (most cited first, 8 of 8)

- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Integrating Large Language Models with Internet of Things Applications](https://mlsanity.com/paper/2410.19223/) (2024)
- [Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens](https://mlsanity.com/paper/2507.02964/) (2025)
- [Constructing Multi-label Hierarchical Classification Models for MITRE ATT&CK Text Tagging](https://mlsanity.com/paper/2601.14556/) (2026)
- [Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports](https://mlsanity.com/paper/2606.18166/) (2026)
- [TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction](https://mlsanity.com/paper/2607.05001/) (2026)
- [RAGRank: Using PageRank to Counter Poisoning in CTI LLM Pipelines](https://mlsanity.com/paper/2510.20768/) (2025)
- [Reasoning Topology Matters: A Controlled Study of LLM-Based Cybersecurity Analysis](https://mlsanity.com/paper/2609.24710/) (2026)

## Abstract (arXiv)

Tactics, Techniques, and Procedures (TTPs) outline the methods attackers use to exploit vulnerabilities. The interpretation of TTPs in the MITRE ATT&CK framework can be challenging for cybersecurity practitioners due to presumed expertise and complex dependencies. Meanwhile, advancements with Large Language Models (LLMs) have led to recent surge in studies exploring its uses in cybersecurity operations. It is, however, unclear how LLMs can be used in an efficient and proper way to provide accurate responses for critical domains such as cybersecurity. This leads us to investigate how to better use two types of LLMs: small-scale encoder-only (e.g., RoBERTa) and larger decoder-only (e.g., GPT-3.5) LLMs to comprehend and summarize TTPs with the intended purposes (i.e., tactics) of a cyberattack procedure. This work studies and compares the uses of supervised fine-tuning (SFT) of encoder-only LLMs vs. Retrieval Augmented Generation (RAG) for decoder-only LLMs (without fine-tuning). Both SFT and RAG techniques presumably enhance the LLMs with relevant contexts for each cyberattack procedure. Our studies show decoder-only LLMs with RAG achieves better performance than encoder-only models with SFT, particularly when directly relevant context is extracted by RAG. The decoder-only results could suffer low `Precision' while achieving high `Recall'. Our findings further highlight a counter-intuitive observation that more generic prompts tend to yield better predictions of cyberattack tactics than those that are more specifically tailored.