# PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)

arXiv: https://arxiv.org/abs/2409.12699 · HTML: https://mlsanity.com/paper/2409.12699/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 75 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [GraphGAN: Graph Representation Learning with Generative Adversarial Nets](https://mlsanity.com/paper/1711.08267/) (2017)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Guiding Large Language Models via Directional Stimulus Prompting](https://mlsanity.com/paper/2302.11520/) (2023)
- [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification](https://mlsanity.com/paper/2305.14752/) (2023)
- [GCN-GAN: A Non-linear Temporal Link Prediction Model for Weighted Dynamic Networks](https://mlsanity.com/paper/1901.09165/) (2019)
- [A large-scale study on research code quality and execution](https://mlsanity.com/paper/2103.12793/) (2021)
- [AixBench: A Code Generation Benchmark Dataset](https://mlsanity.com/paper/2206.13179/) (2022)
- [LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward](https://mlsanity.com/paper/2401.03374/) (2024)

## Built on by (most cited first, 19 of 19)

- [PurpCode: Reasoning for Safer Code Generation](https://mlsanity.com/paper/2507.19060/) (2025)
- [SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories](https://mlsanity.com/paper/2504.21205/) (2025)
- [Training Language Models to Generate Quality Code with Program Analysis Feedback](https://mlsanity.com/paper/2505.22704/) (2025)
- [Rethinking the Evaluation of Secure Code Generation](https://mlsanity.com/paper/2503.15554/) (2025)
- [RESCUE: Retrieval Augmented Secure Code Generation](https://mlsanity.com/paper/2510.18204/) (2025)
- [Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis](https://mlsanity.com/paper/2412.14841/) (2024)
- [Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities](https://mlsanity.com/paper/2509.25894/) (2025)
- [How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test](https://mlsanity.com/paper/2601.07084/) (2026)
- [What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants](https://mlsanity.com/paper/2605.30777/) (2026)
- [PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs](https://mlsanity.com/paper/2509.03117/) (2025)
- [Enhancing Reliability in LLM-Based Secure Code Generation](https://mlsanity.com/paper/2605.24300/) (2026)
- [Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs](https://mlsanity.com/paper/2506.10989/) (2025)
- [Security-by-Design for LLM-Based Code Generation: Leveraging Internal Representations for Concept-Driven Steering Mechanisms](https://mlsanity.com/paper/2603.11212/) (2026)
- [A Quasi-Experimental Developer Study of Security Training in LLM-Assisted Web Application Development](https://mlsanity.com/paper/2604.17763/) (2026)
- [Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration](https://mlsanity.com/paper/2606.05396/) (2026)
- [Activating Latent Security Knowledge through LLM-Guided Risk Analysis for Secure Code Generation](https://mlsanity.com/paper/2606.16244/) (2026)
- [SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward](https://mlsanity.com/paper/2606.25195/) (2026)
- [From Adoption to Deployment: A Qualitative Study on AI Integration in Software Development Practice](https://mlsanity.com/paper/2607.16660/) (2026)
- [The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting](https://mlsanity.com/paper/2607.23710/) (2026)

## Abstract (arXiv)

The capability of generating high-quality source code using large language models (LLMs) reduces software development time and costs. However, they often introduce security vulnerabilities due to training on insecure open-source data. This highlights the need for ensuring secure and functional code generation. This paper introduces PromSec, an algorithm for prom optimization for secure and functioning code generation using LLMs. In PromSec, we combine 1) code vulnerability clearing using a generative adversarial graph neural network, dubbed as gGAN, to fix and reduce security vulnerabilities in generated codes and 2) code generation using an LLM into an interactive loop, such that the outcome of the gGAN drives the LLM with enhanced prompts to generate secure codes while preserving their functionality. Introducing a new contrastive learning approach in gGAN, we formulate code-clearing and generation as a dual-objective optimization problem, enabling PromSec to notably reduce the number of LLM inferences. PromSec offers a cost-effective and practical solution for generating secure, functional code. Extensive experiments conducted on Python and Java code datasets confirm that PromSec effectively enhances code security while upholding its intended functionality. Our experiments show that while a state-of-the-art approach fails to address all code vulnerabilities, PromSec effectively resolves them. Moreover, PromSec achieves more than an order-of-magnitude reduction in operation time, number of LLM queries, and security analysis costs. Furthermore, prompts optimized with PromSec for a certain LLM are transferable to other LLMs across programming languages and generalizable to unseen vulnerabilities in training. This study is a step in enhancing the trustworthiness of LLMs for secure and functional code generation, supporting their integration into real-world software development.