mlsanityData to 7 Oct 2026

ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?

arXiv 2310.09810 ↗First posted 15 Oct 2023cs.SE cs.CRcited by 44

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) like ChatGPT (i.e., gpt-3.5-turbo and gpt-4) exhibited remarkable advancement in a range of software engineering tasks associated with source code such as code review and code generation. In this paper, we undertake a comprehensive study by instructing ChatGPT for four prevalent vulnerability tasks: function and line-level vulnerability prediction, vulnerability classification, severity estimation, and vulnerability repair. We compare ChatGPT with state-of-the-art language models designed for software vulnerability purposes. Through an empirical assessment employing extensive real-world datasets featuring over 190,000 C/C++ functions, we found that ChatGPT achieves limited performance, trailing behind other language models in vulnerability contexts by a significant margin. The experimental outcomes highlight the challenging nature of vulnerability prediction tasks, requiring domain-specific expertise. Despite ChatGPT's substantial model scale, exceeding that of source code-pre-trained language models (e.g., CodeBERT) by a factor of 14,000, the process of fine-tuning remains imperative for ChatGPT to generalize for vulnerability prediction tasks. We publish the studied dataset, experimental prompts for ChatGPT, and experimental results at https://github.com/awsm-research/ChatGPT4Vul.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 16; 2026: 7.

44 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  3. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  4. 2025LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insightscited by 44
  5. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  6. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 39
  7. 2024Multi-role Consensus through LLMs Discussions for Vulnerability Detectioncited by 19
  8. 2024Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detectioncited by 18
  9. 2024To Err is Machine: Vulnerability Detection Challenges LLM Reasoningcited by 17
  10. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  11. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  12. 2024Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattackscited by 15
32 more
  1. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  2. 2024VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Modelscited by 11
  3. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 9
  4. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  5. 2026SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repaircited by 9
  6. 2024Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPTcited by 8
  7. 2025Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Datacited by 8
  8. 2024How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?cited by 7
  9. 2025Agent That Debugs: Dynamic State-Guided Vulnerability Repaircited by 7
  10. 2024Mitigating Data Imbalance for Software Vulnerability Assessment: Does Data Augmentation Help?cited by 6
  11. 2025Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"cited by 5
  12. 2024EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Codecited by 4
  13. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
  14. 2025A Mixture of Linear Corrections Generates Secure Codecited by 4
  15. 2025A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detectioncited by 4
  16. 2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
  17. 2024Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detectioncited by 3
  18. 2025Revisiting Pre-trained Language Models for Vulnerability Detectioncited by 3
  19. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  20. 2024A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicalitycited by 2
  21. 2026AgenticSCR: An Autonomous Agentic Secure Code Review for Immature Vulnerabilities Detectioncited by 2
  22. 2025Worst-Case Symbolic Constraints Analysis and Generalisation with Large Language Modelscited by 1
  23. 2025Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMscited by 1
  24. 2026AgenticRepair: Multi-Faceted Program Context Engineering for Agentic Vulnerability Repaircited by 1
  25. 2024Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localizationcited by 0
  26. 2025Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMscited by 0
  27. 2025Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generationcited by 0
  28. 2025MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detectioncited by 0
  29. 2025On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilitiescited by 0
  30. 2026AutoEG: Exploiting Known Third-Party Vulnerabilities in Black-Box Web Applicationscited by 0
  31. 2026Traffic-Aware Randomized Smoothing for LLM-Based Network Intrusion Detectioncited by 0
  32. 2026On Predicting Vulnerability Severity Using In-Context Learning: An Industrial Case Studycited by 0