Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models
Not on the map. Why?
Abstract (arXiv)
Generative Pre-Trained Transformer models have been shown to be surprisingly effective at a variety of natural language processing tasks -- including generating computer code. We evaluate the effectiveness of open source GPT models for the task of automatic identification of the presence of vulnerable code syntax (specifically targeting C and C++ source code). This task is evaluated on a selection of 36 source code examples from the NIST SARD dataset, which are specifically curated to not contain natural English that indicates the presence, or lack thereof, of a particular vulnerability. The NIST SARD source code dataset contains identified vulnerable lines of source code that are examples of one out of the 839 distinct Common Weakness Enumerations (CWE), allowing for exact quantification of the GPT output classification error rate. A total of 5 GPT models are evaluated, using 10 different inference temperatures and 100 repetitions at each setting, resulting in 5,000 GPT queries per vulnerable source code analyzed. Ultimately, we find that the GPT models that we evaluated are not suitable for fully automated vulnerability scanning because the false positive and false negative rates are too high to likely be useful in practice. However, we do find that the GPT models perform surprisingly well at automated vulnerability detection for some of the test cases, in particular surpassing random sampling, and being able to identify the exact lines of code that are vulnerable albeit at a low success rate. The best performing GPT model result found was Llama-2-70b-chat-hf with inference temperature of 0.1 applied to NIST SARD test case 149165 (which is an example of a buffer overflow vulnerability), which had a binary classification recall score of 1.0 and a precision of 1.0 for correctly and uniquely identifying the vulnerable line of code and the correct CWE number.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 31 Jul 2024.
Builds on
17 of 34 references.
References · 17
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
- 2023Explainability for Large Language Models: A Surveycited by 331
- 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
5 more
- 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
- 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
- 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
- 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
- 2024Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detectioncited by 11
Built on by
Citing papers per year: 2024: 1.
1 paper.