# A systematic literature Review for Transformer-based Software Vulnerability detection

arXiv: https://arxiv.org/abs/2604.24822 · HTML: https://mlsanity.com/paper/2604.24822/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 84 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Large Language Models for Test-Free Fault Localization](https://mlsanity.com/paper/2310.01726/) (2023)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [Pre-training by Predicting Program Dependencies for Vulnerability Analysis Tasks](https://mlsanity.com/paper/2402.00657/) (2024)
- [Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?](https://mlsanity.com/paper/2306.01754/) (2023)
- [SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?](https://mlsanity.com/paper/2307.06616/) (2023)
- [Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT](https://mlsanity.com/paper/2404.17110/) (2024)
- [Harnessing the Power of LLMs in Source Code Vulnerability Detection](https://mlsanity.com/paper/2408.03489/) (2024)
- [An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graph](https://mlsanity.com/paper/2304.11072/) (2023)
- [Line-level Semantic Structure Learning for Code Vulnerability Detection](https://mlsanity.com/paper/2407.18877/) (2024)
- [SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysis](https://mlsanity.com/paper/2508.19472/) (2025)

## Abstract (arXiv)

Context: Software vulnerabilities pose significant security threats to software systems, especially as software is increasingly used across many areas of daily life, including health, government, and finance. Recently, transformer-based models have demonstrated promising results in automatic software vulnerability identification due to their robust contextual modelling and representation learning capabilities. Objectives: While numerous systematic literature reviews (SLRs) have examined machine learning and deep learning methods for identifying vulnerabilities, a more transformer-centric analysis remains to be explored. This SLR critically analysed 80 studies published between 2021 and 2025 that utilised transformer models to identify software vulnerabilities. Methods: Using Kitchenhams SLR guidelines, we methodically evaluate current research from various perspectives, encompassing study trends, datasets and sources, programming languages, transformer frameworks, detection detail levels, assessment metrics, reference models, types of vulnerabilities, and experimental configurations. Results: We classify transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts. The results emphasise prevailing research trends, frequently utilised benchmarks, and main baselines. It also uncovers crucial technical issues like data imbalance, interpretability, scalability, and generalization across programming languages. Conclusion: By integrating current evidence and recognising unaddressed research areas, this SLR provides a consolidated resource for researchers and professionals seeking to develop more reliable, precise, and interpretable transformer-based vulnerability identification systems.