# SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?

arXiv: https://arxiv.org/abs/2307.06616 · HTML: https://mlsanity.com/paper/2307.06616/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 67 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities](https://mlsanity.com/paper/1807.06756/) (2018)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [Neural Transfer Learning for Repairing Security Vulnerabilities in C Code](https://mlsanity.com/paper/2104.08308/) (2021)
- [The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verification](https://mlsanity.com/paper/2307.02192/) (2023)
- [Security Vulnerability Detection Using Deep Learning Natural Language Processing](https://mlsanity.com/paper/2105.02388/) (2021)
- [Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation](https://mlsanity.com/paper/2212.04584/) (2022)

## Built on by (most cited first, 13 of 13)

- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [SECURE: Benchmarking Large Language Models for Cybersecurity](https://mlsanity.com/paper/2405.20441/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery](https://mlsanity.com/paper/2510.14036/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [Advancing LLM-Based Security Automation with Customized Group Relative Policy Optimization for Zero-Touch Networks](https://mlsanity.com/paper/2512.09485/) (2025)
- [EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution](https://mlsanity.com/paper/2605.30105/) (2026)
- [ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?](https://mlsanity.com/paper/2505.11565/) (2025)
- [Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?](https://mlsanity.com/paper/2601.00559/) (2026)
- [DrLLM: Prompt-Enhanced Distributed Denial-of-Service Resistance Method with Large Language Models](https://mlsanity.com/paper/2409.10561/) (2024)
- [A systematic literature Review for Transformer-based Software Vulnerability detection](https://mlsanity.com/paper/2604.24822/) (2026)

## Abstract (arXiv)

Software vulnerabilities can cause numerous problems, including crashes, data loss, and security breaches. These issues greatly compromise quality and can negatively impact the market adoption of software applications and systems. Traditional bug-fixing methods, such as static analysis, often produce false positives. While bounded model checking, a form of Formal Verification (FV), can provide more accurate outcomes compared to static analyzers, it demands substantial resources and significantly hinders developer productivity. Can Machine Learning (ML) achieve accuracy comparable to FV methods and be used in popular instant code completion frameworks in near real-time? In this paper, we introduce SecureFalcon, an innovative model architecture with only 121 million parameters derived from the Falcon-40B model and explicitly tailored for classifying software vulnerabilities. To achieve the best performance, we trained our model using two datasets, namely the FormAI dataset and the FalconVulnDB. The FalconVulnDB is a combination of recent public datasets, namely the SySeVR framework, Draper VDISC, Bigvul, Diversevul, SARD Juliet, and ReVeal datasets. These datasets contain the top 25 most dangerous software weaknesses, such as CWE-119, CWE-120, CWE-476, CWE-122, CWE-190, CWE-121, CWE-78, CWE-787, CWE-20, and CWE-762. SecureFalcon achieves 94% accuracy in binary classification and up to 92% in multiclassification, with instant CPU inference times. It outperforms existing models such as BERT, RoBERTa, CodeBERT, and traditional ML algorithms, promising to push the boundaries of software vulnerability detection and instant code completion frameworks.