mlsanityData to 14 Oct 2026

LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks

arXiv 2312.12575 ↗First posted 19 Dec 2023cs.CRcited by 97

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have been suggested for use in automated vulnerability repair, but benchmarks showing they can consistently identify security-related bugs are lacking. We thus develop SecLLMHolmes, a fully automated evaluation framework that performs the most detailed investigation to date on whether LLMs can reliably identify and reason about security-related bugs. We construct a set of 228 code scenarios and analyze eight of the most capable LLMs across eight different investigative dimensions using our framework. Our evaluation shows LLMs provide non-deterministic responses, incorrect and unfaithful reasoning, and perform poorly in real-world scenarios. Most importantly, our findings reveal significant non-robustness in even the most advanced models like `PaLM2' and `GPT-4': by merely changing function or variable names, or by the addition of library functions in the source code, these models can yield incorrect answers in 26% and 17% of cases, respectively. These findings demonstrate that further LLM advances are needed before LLMs can be used as general purpose security assistants.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Dec 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 44; 2026: 31.

97 papers.

Sort:

By area:

  1. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  2. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
  3. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  4. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
  5. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
  6. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 59
  7. 2025LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insightscited by 47
  8. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  9. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  10. 2024Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detectioncited by 28
  11. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  12. 2025Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agentscited by 20
38 more
  1. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 19
  2. 2024To Err is Machine: Vulnerability Detection Challenges LLM Reasoningcited by 18
  3. 2025LAMD: Context-driven Android Malware Detection and Classification with LLMscited by 18
  4. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 18
  5. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 16
  6. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 14
  7. 2024VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Modelscited by 11
  8. 2024API-guided Dataset Synthesis to Finetune Large Code Modelscited by 11
  9. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  10. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 10
  11. 2025The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generationcited by 10
  12. 2024VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patchingcited by 9
  13. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 9
  14. 2025Automated Profile Inference with Language Model Agentscited by 9
  15. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 9
  16. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agentscited by 8
  17. 2025VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detectioncited by 8
  18. 2024How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?cited by 7
  19. 2025Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overviewcited by 7
  20. 2025R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillationcited by 7
  21. 2024ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMscited by 6
  22. 2024Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysiscited by 6
  23. 2025MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinementcited by 6
  24. 2025One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discoverycited by 6
  25. 2026Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agentscited by 6
  26. 2025Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilitiescited by 5
  27. 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
  28. 2025Trace Gadgets: Minimizing Code Context for Machine Learning-Based Vulnerability Predictioncited by 5
  29. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-benchcited by 5
  30. 2025Automated Penetration Testing with LLM Agents and Classical Planningcited by 5
  31. 2026Beyond OAuth: Task-Scoped Authorization for AI Agents via Natural Language Slicescited by 5
  32. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  33. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  34. 2024Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detectioncited by 4
  35. 2025XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistantscited by 4
  36. 2025AutoPatch: Multi-Agent Framework for Patching Real-World CVE Vulnerabilitiescited by 4
  37. 2025SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflowscited by 4
  38. 2025A Mixture of Linear Corrections Generates Secure Codecited by 4

50 most cited of 97 All 97 (JSON)