mlsanityData to 7 Oct 2026

AttackQA: Development and Adoption of a Dataset for Assisting Cybersecurity Operations using Fine-tuned and Open-Source LLMs

arXiv 2411.01073 ↗First posted 1 Nov 2024cs.LG cs.AI cs.CRcited by 3

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) on specialized domain datasets has shown improved performance when large language models (LLMs) are fine-tuned for generating responses to user queries. In this study, we develop a cybersecurity question-answering (Q&A) dataset, called AttackQA, and employ it to build a RAG-based Q&A system designed for analysts in security operations centers. The dataset comprises 25,335 Q&A pairs, accompanied by rationales to facilitate fine-tuning and evaluation. 80% of the dataset was generated with help of a lightweight open-source LLM (LLama 3 8B), which produced over 1100 tokens per second with full 16-bit precision on SambaNova System's SN40L specialized hardware. To ensure dataset quality, we fine-tuned LLama 3 70B to detect and reject low-quality Q&A pairs. In using the dataset for RAG, we demonstrate that fine-tuning open-source embeddings and LLMs can yield superior accuracy compared to OpenAI's state-of-the-art proprietary embedding and LLM (GPT-4o). Furthermore, we use Llama 3.1 405B as a judge to evaluate answer correctness, enabling the creation of a fully open-source, high-speed RAG and evaluation pipeline with a benchmark for model accuracy.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 10 of its 27 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

10 of 27 references.

References · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  4. 2023Improving Text Embeddings with Large Language Modelscited by 295
  5. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  6. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  7. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  8. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  9. 2023MITRE ATT&CK: State of the Art and Way Forwardcited by 12
  10. 2020Learning the Associations of MITRE ATT&CK Adversarial Techniquescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  2. 2026Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTscited by 1
  3. 2026CyberMaskQA: A Privacy-Aware Benchmark for Evaluating Large Language Models in Cybersecurity Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution
  2. 2024SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
  3. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
  4. 2025Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generation
  5. 2025Large Language Models for Explainable Threat Intelligence
  6. 2025LLM-Assisted Proactive Threat Intelligence for Automated Reasoning
  7. 2026CyberLLM-FINDS 2025: Instruction-Tuned Fine-tuning of Domain-Specific LLMs with Retrieval-Augmented Generation and Graph Integration for MITRE Evaluation
  8. 2025EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering