# SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity arXiv: https://arxiv.org/abs/2412.20787 ยท HTML: https://mlsanity.com/paper/2412.20787/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 7 of its 18 references that are in the map. - First posted on arXiv 30 Dec 2024. ## Builds on (most cited first, 7 of 7) - [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024) - [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024) - [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024) - [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023) - [Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent](https://mlsanity.com/paper/2411.02265/) (2024) - [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge](https://mlsanity.com/paper/2402.07688/) (2024) - [SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security](https://mlsanity.com/paper/2312.15838/) (2023) ## Built on by (most cited first, 21 of 21) - [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025) - [Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report](https://mlsanity.com/paper/2508.01059/) (2025) - [Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges](https://mlsanity.com/paper/2506.17644/) (2025) - [ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation](https://mlsanity.com/paper/2507.14201/) (2025) - [CyberRAG: An Agentic RAG cyber attack classification and reporting tool](https://mlsanity.com/paper/2507.02424/) (2025) - [PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities](https://mlsanity.com/paper/2510.11688/) (2025) - [Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Report](https://mlsanity.com/paper/2601.21051/) (2026) - [AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding](https://mlsanity.com/paper/2503.03170/) (2025) - [Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps](https://mlsanity.com/paper/2604.19533/) (2026) - [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025) - [Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generation](https://mlsanity.com/paper/2510.27080/) (2025) - [CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography](https://mlsanity.com/paper/2512.02625/) (2025) - [RedSage: A Cybersecurity Generalist LLM](https://mlsanity.com/paper/2601.22159/) (2026) - [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026) - [Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents](https://mlsanity.com/paper/2607.15263/) (2026) - [Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response](https://mlsanity.com/paper/2607.25379/) (2026) - [SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response](https://mlsanity.com/paper/2607.26791/) (2026) - [Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities](https://mlsanity.com/paper/2609.15963/) (2026) - [CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices](https://mlsanity.com/paper/2609.21344/) (2026) - [SelfOp: An Optimization Algorithm for Self-Improving Security Agents](https://mlsanity.com/paper/2609.22792/) (2026) - [KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards](https://mlsanity.com/paper/2610.02206/) (2026) ## Abstract (arXiv) Evaluating Large Language Models (LLMs) is crucial for understanding their capabilities and limitations across various applications, including natural language processing and code generation. Existing benchmarks like MMLU, C-Eval, and HumanEval assess general LLM performance but lack focus on specific expert domains such as cybersecurity. Previous attempts to create cybersecurity datasets have faced limitations, including insufficient data volume and a reliance on multiple-choice questions (MCQs). To address these gaps, we propose SecBench, a multi-dimensional benchmarking dataset designed to evaluate LLMs in the cybersecurity domain. SecBench includes questions in various formats (MCQs and short-answer questions (SAQs)), at different capability levels (Knowledge Retention and Logical Reasoning), in multiple languages (Chinese and English), and across various sub-domains. The dataset was constructed by collecting high-quality data from open sources and organizing a Cybersecurity Question Design Contest, resulting in 44,823 MCQs and 3,087 SAQs. Particularly, we used the powerful while cost-effective LLMs to (1). label the data and (2). constructing a grading agent for automatic evaluation of SAQs. Benchmarking results on 16 SOTA LLMs demonstrate the usability of SecBench, which is arguably the largest and most comprehensive benchmark dataset for LLMs in cybersecurity. More information about SecBench can be found at our website, and the dataset can be accessed via the artifact link.