mlsanityData to 7 Oct 2026

How Far Have We Gone in Vulnerability Detection Using Large Language Models

arXiv 2311.12420 ↗First posted 21 Nov 2023cs.AI cs.CL cs.CRcited by 27

Not on the map. Why?

Abstract (arXiv)

As software becomes increasingly complex and prone to vulnerabilities, automated vulnerability detection is critically important, yet challenging. Given the significant successes of large language models (LLMs) in various tasks, there is growing anticipation of their efficacy in vulnerability detection. However, a quantitative understanding of their potential in vulnerability detection is still missing. To bridge this gap, we introduce a comprehensive vulnerability benchmark VulBench. This benchmark aggregates high-quality data from a wide range of CTF (Capture-the-Flag) challenges and real-world applications, with annotations for each vulnerable function detailing the vulnerability type and its root cause. Through our experiments encompassing 16 LLMs and 6 state-of-the-art (SOTA) deep learning-based models and static analyzers, we find that several LLMs outperform traditional deep learning approaches in vulnerability detection, revealing an untapped potential in LLMs. This work contributes to the understanding and utilization of LLMs for enhanced software security.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

33 of 55 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
21 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  3. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  4. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  5. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  6. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  7. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  8. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  9. 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
  10. 2023Prompt-Enhanced Software Vulnerability Detection Using ChatGPTcited by 51
  11. 2023Data Quality for Software Vulnerability Datasetscited by 49
  12. 2021D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysiscited by 48
  13. 2022jTrans: Jump-Aware Transformer for Binary Code Similaritycited by 44
  14. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 44
  15. 2020VulDeeLocator: A Deep Learning-based Fine-grained Vulnerability Detectorcited by 37
  16. 2023Evaluation of ChatGPT Model for Vulnerability Detectioncited by 33
  17. 2017Learn&Fuzz: Machine Learning for Input Fuzzingcited by 23
  18. 2023The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Modelscited by 18
  19. 2020Magma: A Ground-Truth Fuzzing Benchmarkcited by 16
  20. 2023Can Large Language Models Find And Fix Vulnerable Software?cited by 14
  21. 2023Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 10.

27 papers.

Sort:

By area:

  1. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  2. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 67
  3. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  4. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  5. 2025Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineeringcited by 15
  6. 2025Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testingcited by 14
  7. 2024VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Modelscited by 11
  8. 2024Investigating Large Language Models for Code Vulnerability Detection: An Experimental Studycited by 11
  9. 2025Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overviewcited by 7
  10. 2025Reasoning with LLMs for Zero-Shot Vulnerability Detectioncited by 7
  11. 2026Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industrycited by 5
  12. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
15 more
  1. 2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
  2. 2024Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detectioncited by 3
  3. 2025Revisiting Pre-trained Language Models for Vulnerability Detectioncited by 3
  4. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  5. 2025Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detectioncited by 2
  6. 2024Unmasking the Shadows: Pinpoint the Implementations of Anti-Dynamic Analysis Techniques in Malware Using LLMcited by 1
  7. 2026Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detectioncited by 1
  8. 2026Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Softwarecited by 1
  9. 2025MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detectioncited by 0
  10. 2026VulReaD: Knowledge-Graph-guided Software Vulnerability Reasoning and Detectioncited by 0
  11. 2026Seclens: Role-specific Evaluation of LLM's for security vulnerablity detectioncited by 0
  12. 2026OASIF: An Efficient Obfuscation-Aware Self-Improving Framework for LLM-Based Assembly Code Instruction Following and Comprehensioncited by 0
  13. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
  14. 2026Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agendacited by 0
  15. 2026Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Modelscited by 0