# Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models

arXiv: https://arxiv.org/abs/2406.05892 · HTML: https://mlsanity.com/paper/2406.05892/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 26 of 26)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Automated Vulnerability Detection in Source Code Using Deep Representation Learning](https://mlsanity.com/paper/1807.04320/) (2018)
- [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023)
- [Large Language Models for Test-Free Fault Localization](https://mlsanity.com/paper/2310.01726/) (2023)
- [The Art, Science, and Engineering of Fuzzing: A Survey](https://mlsanity.com/paper/1812.00140/) (2018)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [LineVD: Statement-level Vulnerability Detection using Graph Neural Networks](https://mlsanity.com/paper/2203.05181/) (2022)
- [MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning](https://mlsanity.com/paper/2311.02303/) (2023)
- [Maximum Roaming Multi-Task Learning](https://mlsanity.com/paper/2006.09762/) (2020)
- [Toward Improved Deep Learning-based Vulnerability Detection](https://mlsanity.com/paper/2403.03024/) (2024)
- [Your Instructions Are Not Always Helpful: Assessing the Efficacy of Instruction Fine-tuning for Software Vulnerability Detection](https://mlsanity.com/paper/2401.07466/) (2024)
- [PreciseBugCollector: Extensible, Executable and Precise Bug-fix Collection](https://mlsanity.com/paper/2309.06229/) (2023)

## Built on by (most cited first, 14 of 14)

- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation](https://mlsanity.com/paper/2504.04699/) (2025)
- [Evaluating Language Models For Threat Detection in IoT Security Logs](https://mlsanity.com/paper/2507.02390/) (2025)
- [Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization](https://mlsanity.com/paper/2507.03051/) (2025)
- [A Mixture of Linear Corrections Generates Secure Code](https://mlsanity.com/paper/2507.09508/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization](https://mlsanity.com/paper/2511.11896/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection](https://mlsanity.com/paper/2607.13439/) (2026)
- [Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Points](https://mlsanity.com/paper/2412.11194/) (2024)
- [Learning to Triage Vulnerability Reports from Program Analysis: An Empirical Study in Node.js](https://mlsanity.com/paper/2510.20739/) (2025)
- [On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection](https://mlsanity.com/paper/2510.27675/) (2025)
- [An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems](https://mlsanity.com/paper/2601.00254/) (2026)
- [DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localization](https://mlsanity.com/paper/2605.11015/) (2026)

## Abstract (arXiv)

Software security vulnerabilities allow attackers to perform malicious activities to disrupt software operations. Recent Transformer-based language models have significantly advanced vulnerability detection, surpassing the capabilities of static analysis based deep learning models. However, language models trained solely on code tokens do not capture either the explanation of vulnerability type or the data flow structure information of code, both of which are crucial for vulnerability detection. We propose a novel technique that integrates a multitask sequence-to-sequence LLM with pro-gram control flow graphs encoded as a graph neural network to achieve sequence-to-classification vulnerability detection. We introduce MSIVD, multitask self-instructed fine-tuning for vulnerability detection, inspired by chain-of-thought prompting and LLM self-instruction. Our experiments demonstrate that MSIVD achieves superior performance, outperforming the highest LLM-based vulnerability detector baseline (LineVul), with a F1 score of 0.92 on the BigVul dataset, and 0.48 on the PreciseBugs dataset. By training LLMs and GNNs simultaneously using a combination of code and explanatory metrics of a vulnerable program, MSIVD represents a promising direction for advancing LLM-based vulnerability detection that generalizes to unseen data. Based on our findings, we further discuss the necessity for new labelled security vulnerability datasets, as recent LLMs have seen or memorized prior datasets' held-out evaluation data.