mlsanityData to 7 Oct 2026

VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection

arXiv 2205.12424 ↗First posted 25 May 2022cs.CR cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

This paper presents VulBERTa, a deep learning approach to detect security vulnerabilities in source code. Our approach pre-trains a RoBERTa model with a custom tokenisation pipeline on real-world code from open-source C/C++ projects. The model learns a deep knowledge representation of the code syntax and semantics, which we leverage to train vulnerability detection classifiers. We evaluate our approach on binary and multi-class vulnerability detection tasks across several datasets (Vuldeepecker, Draper, REVEAL and muVuldeepecker) and benchmarks (CodeXGLUE and D2A). The evaluation results show that VulBERTa achieves state-of-the-art performance and outperforms existing approaches across different datasets, despite its conceptual simplicity, and limited cost in terms of size of training data and number of model parameters.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

22 of 39 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 361
10 more
  1. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  2. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  3. 2018Automated Vulnerability Detection in Source Code Using Deep Representation Learningcited by 114
  4. 2018SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilitiescited by 91
  5. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  6. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  7. 2021D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysiscited by 48
  8. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  9. 2021On using distributed representations of source code for the detection of C security vulnerabilitiescited by 10
  10. 2021Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 12; 2025: 18; 2026: 12.

51 papers.

Sort:

By area:

  1. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 67
  2. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 44
  3. 2025LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insightscited by 44
  4. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
  5. 2023How Far Have We Gone in Vulnerability Detection Using Large Language Modelscited by 27
  6. 2024LLMDFA: Analyzing Dataflow in Code with Large Language Modelscited by 22
  7. 2022Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detectioncited by 18
  8. 2024Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detectioncited by 18
  9. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  10. 2023SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?cited by 13
  11. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  12. 2023Using ChatGPT as a Static Application Security Testing Toolcited by 8
38 more
  1. 2025Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overviewcited by 7
  2. 2024Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?cited by 6
  3. 2023An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graphcited by 5
  4. 2024A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Taskscited by 4
  5. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  6. 2024EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Codecited by 4
  7. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
  8. 2025A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detectioncited by 4
  9. 2025MalCodeAI: Autonomous Vulnerability Detection and Remediation via Language Agnostic Code Reasoningcited by 3
  10. 2025SecureFixAgent: A Hybrid LLM Agent for Automated Python Static Vulnerability Repaircited by 3
  11. 2026Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Reviewcited by 3
  12. 2026FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproductioncited by 3
  13. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  14. 2025Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detectioncited by 2
  15. 2025SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligencecited by 2
  16. 2025Llama-based source code vulnerability detection: Prompt engineering vs Fine tuningcited by 2
  17. 2023An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Codecited by 1
  18. 2023LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learningcited by 1
  19. 2025Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluationcited by 1
  20. 2026A Dual-Loop Agent Framework for Automated Vulnerability Reproductioncited by 1
  21. 2026PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detectioncited by 1
  22. 2026Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detectioncited by 1
  23. 2024Unintentional Security Flaws in Code: Automated Defense via Root Cause Analysiscited by 0
  24. 2024CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identificationcited by 0
  25. 2024Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Pointscited by 0
  26. 2025Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approachescited by 0
  27. 2025White-Basilisk: A Hybrid Model for Code Vulnerability Detectioncited by 0
  28. 2025Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimizationcited by 0
  29. 2025Semantically Guided Adversarial Testing of Vision Models Using Language Modelscited by 0
  30. 2025LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Codecited by 0
  31. 2025On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detectioncited by 0
  32. 2025From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detectioncited by 0
  33. 2026Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detectioncited by 0
  34. 2026Efficient Software Vulnerability Detection Using Transformer-based Modelscited by 0
  35. 2026A systematic literature Review for Transformer-based Software Vulnerability detectioncited by 0
  36. 2026Learning Generalizable Multimodal Representations for Software Vulnerability Detectioncited by 0
  37. 2026VulStyle: A Multi-Modal Pre-Training for Code Stylometry-Augmented Vulnerability Detectioncited by 0
  38. 2026DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localizationcited by 0

50 most cited of 51 All 51 (JSON)