# Semantic-aware Binary Code Representation with BERT

arXiv: https://arxiv.org/abs/2106.05478 · HTML: https://mlsanity.com/paper/2106.05478/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 74 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [SAFE: Self-Attentive Function Embeddings for Binary Similarity](https://mlsanity.com/paper/1811.05296/) (2018)
- [Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs](https://mlsanity.com/paper/1808.04706/) (2018)
- [Dynamic Malware Analysis with Feature Engineering and Feature Learning](https://mlsanity.com/paper/1907.07352/) (2019)
- [A Survey of Binary Code Similarity](https://mlsanity.com/paper/1909.11424/) (2019)
- [Neurlux: Dynamic Malware Analysis Without Feature Engineering](https://mlsanity.com/paper/1910.11376/) (2019)
- [A Cross-Architecture Instruction Embedding Model for Natural Language Processing-Inspired Binary Code Analysis](https://mlsanity.com/paper/1812.09652/) (2018)
- [SoK: All You Ever Wanted to Know About x86/x64 Binary Disassembly But Were Afraid to Ask](https://mlsanity.com/paper/2007.14266/) (2020)
- [Using Deep Learning to Solve Computer Security Challenges: A Survey](https://mlsanity.com/paper/1912.05721/) (2019)

## Built on by (most cited first, 6 of 6)

- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [UniASM: Binary Code Similarity Detection without Fine-tuning](https://mlsanity.com/paper/2211.01144/) (2022)
- [Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code](https://mlsanity.com/paper/2405.20611/) (2024)
- [Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code](https://mlsanity.com/paper/2409.17513/) (2024)
- [SimCLF: A Simple Contrastive Learning Framework for Function-level Binary Embeddings](https://mlsanity.com/paper/2209.02442/) (2022)

## Abstract (arXiv)

A wide range of binary analysis applications, such as bug discovery, malware analysis and code clone detection, require recovery of contextual meanings on a binary code. Recently, binary analysis techniques based on machine learning have been proposed to automatically reconstruct the code representation of a binary instead of manually crafting specifics of the analysis algorithm. However, the existing approaches utilizing machine learning are still specialized to solve one domain of problems, rendering recreation of models for different types of binary analysis. In this paper, we propose DeepSemantic utilizing BERT in producing the semantic-aware code representation of a binary code. To this end, we introduce well-balanced instruction normalization that holds rich information for each of instructions yet minimizing an out-of-vocabulary (OOV) problem. DeepSemantic has been carefully designed based on our study with large swaths of binaries. Besides, DeepSemantic leverages the essence of the BERT architecture into re-purposing a pre-trained generic model that is readily available as a one-time processing, followed by quickly applying specific downstream tasks with a fine-tuning process. We demonstrate DeepSemantic with two downstream tasks, namely, binary similarity comparison and compiler provenance (i.e., compiler and optimization level) prediction. Our experimental results show that the binary similarity model outperforms two state-of-the-art binary similarity tools, DeepBinDiff and SAFE, 49.84% and 15.83% on average, respectively.