mlsanityData to 7 Oct 2026

Can Large Language Models Find And Fix Vulnerable Software?

arXiv 2308.10345 ↗First posted 20 Aug 2023cs.SE cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

In this study, we evaluated the capability of Large Language Models (LLMs), particularly OpenAI's GPT-4, in detecting software vulnerabilities, comparing their performance against traditional static code analyzers like Snyk and Fortify. Our analysis covered numerous repositories, including those from NASA and the Department of Defense. GPT-4 identified approximately four times the vulnerabilities than its counterparts. Furthermore, it provided viable fixes for each vulnerability, demonstrating a low rate of false positives. Our tests encompassed 129 code samples across eight programming languages, revealing the highest vulnerabilities in PHP and JavaScript. GPT-4's code corrections led to a 90% reduction in vulnerabilities, requiring only an 11% increase in code lines. A critical insight was LLMs' ability to self-audit, suggesting fixes for their identified vulnerabilities and underscoring their precision. Future research should explore system-level vulnerabilities and integrate multiple static code analyzers for a holistic perspective on LLMs' potential.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

16 of 36 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  3. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  4. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  5. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  6. 2023Evaluation of ChatGPT Model for Vulnerability Detectioncited by 33
  7. 2023What do LLMs need to Synthesize Correct Router Configurations?cited by 29
  8. 2023LM4HPC: Towards Effective Language Model Application in High-Performance Computingcited by 18
  9. 2022AI-driven Development Is Here: Should You Worry?cited by 15
  10. 2018A Practical Approach to the Automatic Classification of Security-Relevant Commitscited by 13
  11. 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
  12. 2023A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functionscited by 8
4 more
  1. 2023Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for Familiescited by 5
  2. 2023Unmasking the giant: A comprehensive evaluation of ChatGPT's proficiency in coding algorithms and data structurescited by 3
  3. 2018Toward Validation of Textual Information Retrieval Techniques for Software Weaknessescited by 2
  4. 2023Evaluating GPT's Programming Capability through CodeWars' Katascited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 6; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  3. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 35
  4. 2023How Far Have We Gone in Vulnerability Detection Using Large Language Modelscited by 27
  5. 2023Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptionscited by 20
  6. 2024Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity gamescited by 20
  7. 2025Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineeringcited by 15
  8. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 9
  9. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 8
  10. 2025Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?cited by 2
  11. 2025Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testingcited by 1
  12. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
2 more
  1. 2025Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 1
  2. 2026Security of LLM-generated Code: A Comparative Analysiscited by 0