mlsanityData to 14 Oct 2026

SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security

arXiv 2312.15838 ↗First posted 26 Dec 2023cs.CL cs.CRcited by 37

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce SecQA, a novel dataset tailored for evaluating the performance of Large Language Models (LLMs) in the domain of computer security. Utilizing multiple-choice questions generated by GPT-4 based on the "Computer Systems Security: Planning for Success" textbook, SecQA aims to assess LLMs' understanding and application of security principles. We detail the structure and intent of SecQA, which includes two versions of increasing complexity, to provide a concise evaluation across various difficulty levels. Additionally, we present an extensive evaluation of prominent LLMs, including GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral, and Zephyr models, using both 0-shot and 5-shot learning settings. Our results, encapsulated in the SecQA v1 and v2 datasets, highlight the varying capabilities and limitations of these models in the computer security context. This study not only offers insights into the current state of LLMs in understanding security-related content but also establishes SecQA as a benchmark for future advancements in this critical research area.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 26 Dec 2023.

Builds on

9 of 16 references.

References · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  6. 2021V2W-BERT: A Framework for Effective Hierarchical Multiclass Classification of Software Vulnerabilitiescited by 13
  7. 2021MalBERT: Using Transformers for Cybersecurity and Malicious Software Detectioncited by 12
  8. 2021CVSS-BERT: Explainable Natural Language Processing to Determine the Severity of a Computer Security Vulnerability from its Descriptioncited by 10
  9. 2023Anomaly Detection of Command Shell Sessions based on DistilBERT: Unsupervised and Supervised Approachescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 19; 2026: 7.

37 papers.

Sort:

By area:

  1. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
  2. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  3. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
  4. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  5. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 23
  6. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  7. 2024SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecuritycited by 21
  8. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 16
  9. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  10. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  11. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 14
  12. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
25 more
  1. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 8
  2. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  3. 2024TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questionscited by 7
  4. 2024Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Modelscited by 7
  5. 2024Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidencecited by 5
  6. 2024Multi-Agent Collaboration in Incident Response with Large Language Modelscited by 4
  7. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 4
  8. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 4
  9. 2025CyberLLMInstruct: A Pseudo-malicious Dataset Revealing Safety-performance Trade-offs in Cyber Security LLM Fine-tuningcited by 3
  10. 2025AutoBnB-RAG: Enhancing Multi-Agent Incident Response with Retrieval-Augmented Generationcited by 3
  11. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformationscited by 3
  12. 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agentscited by 3
  13. 2025MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarkscited by 2
  14. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corporacited by 2
  15. 2025HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilitiescited by 2
  16. 2025EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answeringcited by 2
  17. 2025CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptographycited by 1
  18. 2026CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledgecited by 1
  19. 2024LISAA: A Framework for Large Language Model Information Security Awareness Assessmentcited by 0
  20. 2025Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMscited by 0
  21. 2025Assessing the Software Security Comprehension of Large Language Modelscited by 0
  22. 2026CyberMaskQA: A Privacy-Aware Benchmark for Evaluating Large Language Models in Cybersecurity Question Answeringcited by 0
  23. 2026Domyn-Small: A European 10B Reasoning Language Modelcited by 0
  24. 2026CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wildcited by 0
  25. 2026LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluationcited by 0