# Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code

arXiv: https://arxiv.org/abs/2409.17513 · HTML: https://mlsanity.com/paper/2409.17513/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024)
- [Finetuning Large Language Models for Vulnerability Detection](https://mlsanity.com/paper/2401.17010/) (2024)
- [Transformer-Based Language Models for Software Vulnerability Detection](https://mlsanity.com/paper/2204.03214/) (2022)
- [Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning](https://mlsanity.com/paper/2406.03718/) (2024)
- [Security Vulnerability Detection Using Deep Learning Natural Language Processing](https://mlsanity.com/paper/2105.02388/) (2021)
- [VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models](https://mlsanity.com/paper/2406.07595/) (2024)
- [Evaluating Large Language Models for Line-Level Vulnerability Localization](https://mlsanity.com/paper/2404.00287/) (2024)
- [Semantic-aware Binary Code Representation with BERT](https://mlsanity.com/paper/2106.05478/) (2021)
- [Deep-Learning-based Vulnerability Detection in Binary Executables](https://mlsanity.com/paper/2212.01254/) (2022)
- [An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graph](https://mlsanity.com/paper/2304.11072/) (2023)
- [Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code](https://mlsanity.com/paper/2405.20611/) (2024)

## Built on by (most cited first, 1 of 1)

- [Impact of Data Snooping on Deep Learning Models for Locating Vulnerabilities in Lifted Code](https://mlsanity.com/paper/2412.02048/) (2024)

## Abstract (arXiv)

Ransomware and other forms of malware cause significant financial and operational damage to organizations by exploiting long-standing and often difficult-to-detect software vulnerabilities. To detect vulnerabilities such as buffer overflows in compiled code, this research investigates the application of unidirectional transformer-based embeddings, specifically GPT-2. Using a dataset of LLVM functions, we trained a GPT-2 model to generate embeddings, which were subsequently used to build LSTM neural networks to differentiate between vulnerable and non-vulnerable code. Our study reveals that embeddings from the GPT-2 model significantly outperform those from bidirectional models of BERT and RoBERTa, achieving an accuracy of 92.5% and an F1-score of 89.7%. LSTM neural networks were developed with both frozen and unfrozen embedding model layers. The model with the highest performance was achieved when the embedding layers were unfrozen. Further, the research finds that, in exploring the impact of different optimizers within this domain, the SGD optimizer demonstrates superior performance over Adam. Overall, these findings reveal important insights into the potential of unidirectional transformer-based approaches in enhancing cybersecurity defenses.