# BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer

arXiv: https://arxiv.org/abs/2208.06692 · HTML: https://mlsanity.com/paper/2208.06692/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 13 Aug 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity](https://mlsanity.com/paper/2012.08680/) (2020)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs](https://mlsanity.com/paper/1808.04706/) (2018)
- [How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection](https://mlsanity.com/paper/1902.10339/) (2019)
- [BinMLM: Binary Authorship Verification with Flow-aware Mixture-of-Shared Language Model](https://mlsanity.com/paper/2203.04472/) (2022)
- [In Nomine Function: Naming Functions in Stripped Binaries with Neural Networks](https://mlsanity.com/paper/1912.07946/) (2019)

## Built on by (most cited first, 8 of 8)

- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [kTrans: Knowledge-Aware Transformer for Binary Code Embedding](https://mlsanity.com/paper/2308.12659/) (2023)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding](https://mlsanity.com/paper/2409.07889/) (2024)
- [On the Lack of Robustness of Binary Function Similarity Systems](https://mlsanity.com/paper/2412.04163/) (2024)
- [A Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer](https://mlsanity.com/paper/2412.11177/) (2024)
- [REx86: A Local Large Language Model for Assisting in x86 Assembly Reverse Engineering](https://mlsanity.com/paper/2510.20975/) (2025)
- [SoK: AI-Augmented Binary Reversing](https://mlsanity.com/paper/2606.17398/) (2026)

## Abstract (arXiv)

A recent trend in binary code analysis promotes the use of neural solutions based on instruction embedding models. An instruction embedding model is a neural network that transforms sequences of assembly instructions into embedding vectors. If the embedding network is trained such that the translation from code to vectors partially preserves the semantic, the network effectively represents an assembly code model. In this paper we present BinBert, a novel assembly code model. BinBert is built on a transformer pre-trained on a huge dataset of both assembly instruction sequences and symbolic execution information. BinBert can be applied to assembly instructions sequences and it is fine-tunable, i.e. it can be re-trained as part of a neural architecture on task-specific data. Through fine-tuning, BinBert learns how to apply the general knowledge acquired with pre-training to the specific task. We evaluated BinBert on a multi-task benchmark that we specifically designed to test the understanding of assembly code. The benchmark is composed of several tasks, some taken from the literature, and a few novel tasks that we designed, with a mix of intrinsic and downstream tasks. Our results show that BinBert outperforms state-of-the-art models for binary instruction embedding, raising the bar for binary code understanding.