mlsanityData to 7 Oct 2026

CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence

arXiv 2406.07599 ↗First posted 11 Jun 2024cs.CR cs.AIcited by 61

Not on the map. Why?

Abstract (arXiv)

Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark designed to assess LLMs' performance in CTI applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art models on these tasks provides insights into their strengths and weaknesses in CTI contexts, contributing to a better understanding of LLM capabilities in CTI.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 17 of its 50 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

17 of 50 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  10. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  11. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  12. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
5 more
  1. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 94
  2. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  3. 2023OWL: A Large Language Model for IT Operationscited by 43
  4. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 39
  5. 2023SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Securitycited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 27; 2026: 31.

61 papers.

Sort:

By area:

  1. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  2. 2026Qwen3-Coder-Next Technical Reportcited by 60
  3. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  4. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  5. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  6. 2026ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agentscited by 16
  7. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 15
  8. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  9. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  10. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoningcited by 12
  11. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agentscited by 11
  12. 2026Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenderscited by 10
38 more
  1. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Textcited by 9
  2. 2025SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysiscited by 8
  3. 2025Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitationcited by 4
  4. 2025DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Responsecited by 4
  5. 2025DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environmentscited by 4
  6. 2025Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokenscited by 4
  7. 2025Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Huntingcited by 4
  8. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 4
  9. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 4
  10. 2025Auspex: Building Threat Modeling Tradecraft into an Artificial Intelligence-based Copilotcited by 3
  11. 2025Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applicationscited by 3
  12. 2025Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligencecited by 3
  13. 2025Toward Cybersecurity-Expert Small Language Modelscited by 3
  14. 2026Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibilitycited by 3
  15. 2024Evaluating the Usability of LLMs in Threat Intelligence Enrichmentcited by 2
  16. 2025Military AI Cyber Agents (MAICAs) Constitute a Global Threat to Critical Infrastructurecited by 2
  17. 2025False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systemscited by 2
  18. 2025Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunitiescited by 2
  19. 2025Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generationcited by 2
  20. 2026Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMscited by 2
  21. 2026CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?cited by 2
  22. 2026CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilitiescited by 2
  23. 2026Antares: Foundation Models for Agentic Vulnerability Localizationcited by 2
  24. 2025ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?cited by 1
  25. 2026Constructing Multi-label Hierarchical Classification Models for MITRE ATT&CK Text Taggingcited by 1
  26. 2026OpenSec: Measuring Incident Response Agent Calibration Under Adversarial Evidencecited by 1
  27. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
  28. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  29. 2026CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledgecited by 1
  30. 2026Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoningcited by 1
  31. 2026Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reportscited by 1
  32. 2026FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelinescited by 1
  33. 2026Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysiscited by 1
  34. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  35. 2025ForensicsData: A Digital Forensics Dataset for Large Language Modelscited by 0
  36. 2025Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMscited by 0
  37. 2025RAGRank: Using PageRank to Counter Poisoning in CTI LLM Pipelinescited by 0
  38. 2025From Description to Score: Can LLMs Quantify Vulnerabilities?cited by 0

50 most cited of 61 All 61 (JSON)