# Pre-training by Predicting Program Dependencies for Vulnerability Analysis Tasks

arXiv: https://arxiv.org/abs/2402.00657 · HTML: https://mlsanity.com/paper/2402.00657/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2024.

## Built on by (most cited first, 17 of 17)

- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection](https://mlsanity.com/paper/2407.16235/) (2024)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement](https://mlsanity.com/paper/2510.00317/) (2025)
- [A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection](https://mlsanity.com/paper/2512.16538/) (2025)
- [LLM-based Vulnerability Discovery through the Lens of Code Metrics](https://mlsanity.com/paper/2509.19117/) (2025)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization](https://mlsanity.com/paper/2511.11896/) (2025)
- [VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.09461/) (2026)
- [MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection](https://mlsanity.com/paper/2510.04397/) (2025)
- [Diverse LLMs vs. Vulnerabilities: Who Detects and Fixes Them Better?](https://mlsanity.com/paper/2512.12536/) (2025)
- [A systematic literature Review for Transformer-based Software Vulnerability detection](https://mlsanity.com/paper/2604.24822/) (2026)
- [DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localization](https://mlsanity.com/paper/2605.11015/) (2026)
- [JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode](https://mlsanity.com/paper/2607.02825/) (2026)
- [SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training](https://mlsanity.com/paper/2607.17139/) (2026)
- [JevVibe: Efficient Classification-Guided Secure Code Generation](https://mlsanity.com/paper/2609.34963/) (2026)

## Abstract (arXiv)

Vulnerability analysis is crucial for software security. This work focuses on using pre-training techniques to enhance the understanding of vulnerable code and boost vulnerability analysis. The code understanding ability of a pre-trained model is highly related to its pre-training objectives. The semantic structure, e.g., control and data dependencies, of code is important for vulnerability analysis. However, existing pre-training objectives either ignore such structure or focus on learning to use it. The feasibility and benefits of learning the knowledge of analyzing semantic structure have not been investigated. To this end, this work proposes two novel pre-training objectives, namely Control Dependency Prediction (CDP) and Data Dependency Prediction (DDP), which aim to predict the statement-level control dependencies and token-level data dependencies, respectively, in a code snippet only based on its source code. During pre-training, CDP and DDP can guide the model to learn the knowledge required for analyzing fine-grained dependencies in code. After pre-training, the pre-trained model can boost the understanding of vulnerable code during fine-tuning and can directly be used to perform dependence analysis for both partial and complete functions. To demonstrate the benefits of our pre-training objectives, we pre-train a Transformer model named PDBERT with CDP and DDP, fine-tune it on three vulnerability analysis tasks, i.e., vulnerability detection, vulnerability classification, and vulnerability assessment, and also evaluate it on program dependence analysis. Experimental results show that PDBERT benefits from CDP and DDP, leading to state-of-the-art performance on the three downstream tasks. Also, PDBERT achieves F1-scores of over 99% and 94% for predicting control and data dependencies, respectively, in partial and complete functions.