mlsanityData to 7 Oct 2026

Examining Zero-Shot Vulnerability Repair with Large Language Models

arXiv 2112.02125 ↗First posted 3 Dec 2021cs.CR cs.AIcited by 105

Not on the map. Why?

Abstract (arXiv)

Human developers can produce code with cybersecurity bugs. Can emerging 'smart' code completion tools help repair those bugs? In this work, we examine the use of large language models (LLMs) for code (such as OpenAI's Codex and AI21's Jurassic J-1) for zero-shot vulnerability repair. We investigate challenges in the design of prompts that coax LLMs into generating repaired versions of insecure code. This is difficult due to the numerous ways to phrase key information - both semantically and syntactically - with natural languages. We perform a large scale study of five commercially available, black-box, "off-the-shelf" LLMs, as well as an open-source model and our own locally-trained model, on a mix of synthetic, hand-crafted, and real-world security bug scenarios. Our experiments demonstrate that while the approach has promise (the LLMs could collectively repair 100% of our synthetically generated and hand-crafted scenarios), a qualitative evaluation of the model's performance over a corpus of historical real-world examples highlights challenges in generating functionally correct code.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 8 of its 56 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

8 of 56 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  5. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  6. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  7. 2018Automatic Software Repair: a Bibliographycited by 24
  8. 2021Generating Bug-Fixes Using Pretrained Transformerscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 35; 2025: 23; 2026: 16.

105 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  3. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 136
  4. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  5. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  6. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  7. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  8. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  9. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
  10. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  11. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
  12. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 58
38 more
  1. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  2. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  3. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 50
  4. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 50
  5. 2023Large Language Models and Simple, Stupid Bugscited by 45
  6. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  7. 2023A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verificationcited by 42
  8. 2024PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generationcited by 36
  9. 2023Large Language Models Are Zero-Shot Text Classifierscited by 35
  10. 2024Constrained Decoding for Secure Code Generationcited by 35
  11. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  12. 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testingcited by 34
  13. 2023The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verificationcited by 32
  14. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  15. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  16. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  17. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  18. 2023Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptionscited by 20
  19. 2024LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systemscited by 20
  20. 2024Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity gamescited by 20
  21. 2023Exploring ChatGPT's Capabilities on Vulnerability Managementcited by 19
  22. 2023The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Modelscited by 18
  23. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 18
  24. 2023Out of the Cage: How Stochastic Parrots Win in Cyber Security Environmentscited by 17
  25. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  26. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  27. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  28. 2023RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shotcited by 14
  29. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  30. 2024CigaR: Cost-efficient Program Repair with LLMscited by 14
  31. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  32. 2023Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domaincited by 8
  33. 2023HW-V2W-Map: Hardware Vulnerability to Weakness Mapping Framework for Root Cause Analysis with GPT-assisted Mitigation Suggestioncited by 8
  34. 2024A Survey of Large Language Models in Cybersecuritycited by 8
  35. 2024Location is Key: Leveraging Large Language Model for Functional Bug Localization in Verilogcited by 8
  36. 2023FLAG: Finding Line Anomalies (in code) with Generative AIcited by 7
  37. 2024Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugscited by 7
  38. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 6

50 most cited of 105 All 105 (JSON)