mlsanityData to 14 Oct 2026

Evaluation of ChatGPT Model for Vulnerability Detection

arXiv 2304.07232 ↗First posted 12 Apr 2023cs.CR cs.AI cs.SEcited by 33

Not on the map. Why?

Abstract (arXiv)

In this technical report, we evaluated the performance of the ChatGPT and GPT-3 models for the task of vulnerability detection in code. Our evaluation was conducted on our real-world dataset, using binary and multi-label classification tasks on CWE vulnerabilities. We decided to evaluate the model because it has shown good performance on other code-based tasks, such as solving programming challenges and understanding code at a high level. However, we found that the ChatGPT model performed no better than a dummy classifier for both binary and multi-label classification tasks for code vulnerability detection.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 6 of its 12 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

6 of 12 references.

References · 6

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  6. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 14; 2025: 6; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  3. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
  4. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 77
  5. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  6. 2023GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysiscited by 65
  7. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 40
  8. 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 33
  9. 2024Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuningcited by 30
  10. 2023How Far Have We Gone in Vulnerability Detection Using Large Language Modelscited by 27
  11. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  12. 2023Can Large Language Models Find And Fix Vulnerable Software?cited by 15
21 more
  1. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  2. 2024Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awarenesscited by 14
  3. 2024LLbezpeky: Leveraging Large Language Models for Vulnerability Detectioncited by 13
  4. 2024GenKubeSec: LLM-Based Kubernetes Misconfiguration Detection, Localization, Reasoning, and Remediationcited by 11
  5. 2024Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPTcited by 9
  6. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 9
  7. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  8. 2023A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functionscited by 8
  9. 2023Using ChatGPT as a Static Application Security Testing Toolcited by 8
  10. 2025Agent That Debugs: Dynamic State-Guided Vulnerability Repaircited by 7
  11. 2023Exploring the Limits of ChatGPT in Software Security Applicationscited by 6
  12. 2024ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMscited by 6
  13. 2024Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluationcited by 5
  14. 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
  15. 2023How Does Naming Affect LLMs on Code Analysis Tasks?cited by 4
  16. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  17. 2023Evaluating GPT's Programming Capability through CodeWars' Katascited by 2
  18. 2025LLM Embedding-based Attribution (LEA): Quantifying Source Contributions to Generative Model's Response for Vulnerability Analysiscited by 2
  19. 2024Unveiling Hidden Links Between Unseen Security Entitiescited by 0
  20. 2025An Initial Exploration of Fine-tuning Small Language Models for Smart Contract Reentrancy Vulnerability Detectioncited by 0
  21. 2026QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testingcited by 0