# Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning

arXiv: https://arxiv.org/abs/2406.03718 · HTML: https://mlsanity.com/paper/2406.03718/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning](https://mlsanity.com/paper/2308.00436/) (2023)
- [Deductive Verification of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2306.03872/) (2023)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities](https://mlsanity.com/paper/1807.06756/) (2018)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [Data Quality for Software Vulnerability Datasets](https://mlsanity.com/paper/2301.05456/) (2023)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [Evaluation of ChatGPT Model for Vulnerability Detection](https://mlsanity.com/paper/2304.07232/) (2023)
- [ReGVD: Revisiting Graph Neural Networks for Vulnerability Detection](https://mlsanity.com/paper/2110.07317/) (2021)
- [Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions](https://mlsanity.com/paper/2311.13982/) (2023)
- [An Extensive Study on Adversarial Attack against Pre-trained Models of Code](https://mlsanity.com/paper/2311.07553/) (2023)

## Built on by (most cited first, 30 of 30)

- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [An Empirical Study on LLM-based Agents for Automated Bug Fixing](https://mlsanity.com/paper/2411.10213/) (2024)
- [Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study](https://mlsanity.com/paper/2412.18260/) (2024)
- [VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection](https://mlsanity.com/paper/2512.07533/) (2025)
- [Multi-Programming Language Sandbox for LLMs](https://mlsanity.com/paper/2410.23074/) (2024)
- [Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences](https://mlsanity.com/paper/2506.00419/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation](https://mlsanity.com/paper/2504.04699/) (2025)
- [Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence](https://mlsanity.com/paper/2509.23573/) (2025)
- [Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model](https://mlsanity.com/paper/2602.07422/) (2026)
- [A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat](https://mlsanity.com/paper/2507.18515/) (2025)
- [Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection](https://mlsanity.com/paper/2412.12039/) (2024)
- [A Mixture of Linear Corrections Generates Secure Code](https://mlsanity.com/paper/2507.09508/) (2025)
- [CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence](https://mlsanity.com/paper/2510.11974/) (2025)
- [VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications](https://mlsanity.com/paper/2511.04014/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [Llama-based source code vulnerability detection: Prompt engineering vs Fine tuning](https://mlsanity.com/paper/2512.09006/) (2025)
- [Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code](https://mlsanity.com/paper/2409.17513/) (2024)
- [Large Language Model based Smart Contract Auditing with LLMBugScanner](https://mlsanity.com/paper/2512.02069/) (2025)
- [MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection](https://mlsanity.com/paper/2510.04397/) (2025)
- [On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection](https://mlsanity.com/paper/2510.27675/) (2025)
- [On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities](https://mlsanity.com/paper/2512.20062/) (2025)
- [Learning Generalizable Multimodal Representations for Software Vulnerability Detection](https://mlsanity.com/paper/2604.25711/) (2026)
- [DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localization](https://mlsanity.com/paper/2605.11015/) (2026)
- [Security of LLM-generated Code: A Comparative Analysis](https://mlsanity.com/paper/2605.23091/) (2026)
- [A Unified Model for Cross-Domain Clone Detection via Model Merging](https://mlsanity.com/paper/2608.04215/) (2026)

## Abstract (arXiv)

Code Pre-trained Models (CodePTMs) based vulnerability detection have achieved promising results over recent years. However, these models struggle to generalize as they typically learn superficial mapping from source code to labels instead of understanding the root causes of code vulnerabilities, resulting in poor performance in real-world scenarios beyond the training instances. To tackle this challenge, we introduce VulLLM, a novel framework that integrates multi-task learning with Large Language Models (LLMs) to effectively mine deep-seated vulnerability features. Specifically, we construct two auxiliary tasks beyond the vulnerability detection task. First, we utilize the vulnerability patches to construct a vulnerability localization task. Second, based on the vulnerability features extracted from patches, we leverage GPT-4 to construct a vulnerability interpretation task. VulLLM innovatively augments vulnerability classification by leveraging generative LLMs to understand complex vulnerability patterns, thus compelling the model to capture the root causes of vulnerabilities rather than overfitting to spurious features of a single task. The experiments conducted on six large datasets demonstrate that VulLLM surpasses seven state-of-the-art models in terms of effectiveness, generalization, and robustness.