mlsanityData to 14 Oct 2026

Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild

arXiv 2307.10214 ↗First posted 14 Jul 2023cs.CR cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Cyber Threat Intelligence (CTI) plays a crucial role in assessing risks and enhancing security for organizations. However, the process of extracting relevant information from unstructured text sources can be expensive and time-consuming. Our empirical experience shows that existing tools for automated structured CTI extraction have performance limitations. Furthermore, the community lacks a common benchmark to quantitatively assess their performance. We fill these gaps providing a new large open benchmark dataset and aCTIon, a structured CTI information extraction tool. The dataset includes 204 real-world publicly available reports and their corresponding structured CTI information in STIX format. Our team curated the dataset involving three independent groups of CTI analysts working over the course of several months. To the best of our knowledge, this dataset is two orders of magnitude larger than previously released open source datasets. We then design aCTIon, leveraging recently introduced large language models (GPT3.5) in the context of two custom information extraction pipelines. We compare our method with 10 solutions presented in previous work, for which we develop our own implementations when open-source implementations were lacking. Our results show that aCTIon outperforms previous work for structured CTI extraction with an improvement of the F1-score from 10%points to 50%points across all tasks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 32 of its 63 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

32 of 63 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
20 more
  1. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 932
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  3. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  5. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  6. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  7. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 73
  8. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  9. 2020FLERT: Document-Level Features for Named Entity Recognitioncited by 45
  10. 2021AttacKG: Constructing Technique Knowledge Graph from Cyber Threat Intelligence Reportscited by 31
  11. 2019POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Huntingcited by 27
  12. 2022Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTIcited by 27
  13. 2021EXTRACTOR: Extracting Attack Behavior from Threat Reportscited by 19
  14. 2019GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognitioncited by 18
  15. 2020Automated Retrieval of ATT&CK Tactics and Techniques for Cyber Threat Reportscited by 17
  16. 2020Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligencecited by 12
  17. 2022Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Studycited by 11
  18. 2022ThreatKG: An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Managementcited by 9
  19. 2021A System for Automated Open-Source Threat Intelligence Gathering and Managementcited by 7
  20. 2020Learning the Associations of MITRE ATT&CK Adversarial Techniquescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 5; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  2. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  3. 2024LLMCloudHunter: Harnessing LLMs for Automated Extraction of Detection Rules from Cloud-Based CTIcited by 18
  4. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  5. 2024Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Modelscited by 14
  6. 2024CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Modelscited by 14
  7. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  8. 2024AI-Driven Cyber Threat Intelligence Automationcited by 2
  9. 2025Towards AI-Driven Human-Machine Co-Teaming for Adaptive and Agile Cyber Security Operation Centerscited by 2
  10. 2026CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?cited by 2
  11. 2026TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Constructioncited by 1
  12. 2025Large Language Models for Explainable Threat Intelligencecited by 0
3 more
  1. 2025CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMscited by 0
  2. 2026Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilitiescited by 0
  3. 2026SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMscited by 0