# Large Language Model for Vulnerability Detection: Emerging Results and Future Directions

arXiv: https://arxiv.org/abs/2401.15468 · HTML: https://mlsanity.com/paper/2401.15468/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2024.

## Built on by (most cited first, 30 of 45)

- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents](https://mlsanity.com/paper/2505.10961/) (2025)
- [Multi-role Consensus through LLMs Discussions for Vulnerability Detection](https://mlsanity.com/paper/2403.14274/) (2024)
- [Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study](https://mlsanity.com/paper/2405.15614/) (2024)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study](https://mlsanity.com/paper/2412.18260/) (2024)
- [Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies](https://mlsanity.com/paper/2510.25506/) (2025)
- [VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models](https://mlsanity.com/paper/2406.07595/) (2024)
- [Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions](https://mlsanity.com/paper/2502.14202/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM](https://mlsanity.com/paper/2512.01356/) (2025)
- [Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms](https://mlsanity.com/paper/2507.06323/) (2025)
- [MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement](https://mlsanity.com/paper/2510.00317/) (2025)
- [Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents](https://mlsanity.com/paper/2602.02164/) (2026)
- [CoDefeater: Using LLMs To Find Defeaters in Assurance Cases](https://mlsanity.com/paper/2407.13717/) (2024)
- [Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"](https://mlsanity.com/paper/2502.06898/) (2025)
- [Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap](https://mlsanity.com/paper/2512.04680/) (2025)
- [A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection](https://mlsanity.com/paper/2512.16538/) (2025)
- [Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry](https://mlsanity.com/paper/2601.18844/) (2026)
- [Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering](https://mlsanity.com/paper/2601.22952/) (2026)
- [LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs](https://mlsanity.com/paper/2505.24451/) (2025)
- [A Mixture of Linear Corrections Generates Secure Code](https://mlsanity.com/paper/2507.09508/) (2025)
- [OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Models](https://mlsanity.com/paper/2510.01409/) (2025)
- [VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications](https://mlsanity.com/paper/2511.04014/) (2025)
- [AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection](https://mlsanity.com/paper/2604.11950/) (2026)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review](https://mlsanity.com/paper/2603.18740/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)

## Abstract (arXiv)

Previous learning-based vulnerability detection methods relied on either medium-sized pre-trained models or smaller neural networks from scratch. Recent advancements in Large Pre-Trained Language Models (LLMs) have showcased remarkable few-shot learning capabilities in various tasks. However, the effectiveness of LLMs in detecting software vulnerabilities is largely unexplored. This paper aims to bridge this gap by exploring how LLMs perform with various prompts, particularly focusing on two state-of-the-art LLMs: GPT-3.5 and GPT-4. Our experimental results showed that GPT-3.5 achieves competitive performance with the prior state-of-the-art vulnerability detection approach and GPT-4 consistently outperformed the state-of-the-art.