mlsanityData to 14 Oct 2026

SECURE: Benchmarking Large Language Models for Cybersecurity

arXiv 2405.20441 ↗First posted 30 May 2024cs.CR cs.AI cs.HCcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness. Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks. To address this gap, we introduce the SECURE (Security Extraction, Understanding & Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios. SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources. Our study evaluates seven state-of-the-art models on these tasks, providing insights into their strengths and weaknesses in cybersecurity contexts, and offer recommendations for improving LLMs reliability as cyber advisory tools.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 35 of its 70 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

35 of 70 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
23 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  6. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  7. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  8. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  9. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 128
  10. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  11. 2024IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilitiescited by 103
  12. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 97
  13. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  14. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 78
  15. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  16. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  17. 2023OWL: A Large Language Model for IT Operationscited by 43
  18. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 40
  19. 2023SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Securitycited by 37
  20. 2022CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domaincited by 31
  21. 2023SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?cited by 13
  22. 2023ChatGPT and Other Large Language Models for Cybersecurity of Smart Grid Applicationscited by 13
  23. 2023An Empirical Study of NetOps Capability of Pre-Trained Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 31
  2. 2025AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 4
  3. 2025Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applicationscited by 3
  4. 2026Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibilitycited by 3
  5. 2025MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarkscited by 2
  6. 2025Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunitiescited by 2
  7. 2025Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generationcited by 2
  8. 2026Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMscited by 2
  9. 2026CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?cited by 2
  10. 2026NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Roomscited by 2
  11. 2026RedSage: A Cybersecurity Generalist LLMcited by 1
  12. 2026RuleForge: Automated Generation and Validation for Web Vulnerability Detection at Scalecited by 1
2 more
  1. 2025Assessing the Software Security Comprehension of Large Language Modelscited by 0
  2. 2026Agentic-ZTA: A Multi-Agent Architecture for Autonomous Zero Trust Enforcementcited by 0