# kTrans: Knowledge-Aware Transformer for Binary Code Embedding

arXiv: https://arxiv.org/abs/2308.12659 · HTML: https://mlsanity.com/paper/2308.12659/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [A Mutual Information Maximization Perspective of Language Representation Learning](https://mlsanity.com/paper/1910.08350/) (2019)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [DeepBugs: A Learning Approach to Name-based Bug Detection](https://mlsanity.com/paper/1805.11683/) (2018)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [jTrans: Jump-Aware Transformer for Binary Code Similarity](https://mlsanity.com/paper/2205.12713/) (2022)
- [SAFE: Self-Attentive Function Embeddings for Binary Similarity](https://mlsanity.com/paper/1811.05296/) (2018)
- [XDA: Accurate, Robust Disassembly with Transfer Learning](https://mlsanity.com/paper/2010.00770/) (2020)
- [Revisiting Binary Code Similarity Analysis using Interpretable Feature Engineering and Lessons Learned](https://mlsanity.com/paper/2011.10749/) (2020)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [Pre-Training Representations of Binary Code Using Contrastive Learning](https://mlsanity.com/paper/2210.05102/) (2022)
- [Callee: Recovering Call Graphs for Binaries with Transfer and Contrastive Learning](https://mlsanity.com/paper/2111.01415/) (2021)

## Built on by (most cited first, 10 of 10)

- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision](https://mlsanity.com/paper/2402.16928/) (2024)
- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking](https://mlsanity.com/paper/2402.11842/) (2024)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [VEXIR2Vec: An Architecture-Neutral Embedding Framework for Binary Similarity](https://mlsanity.com/paper/2312.00507/) (2023)
- [A Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer](https://mlsanity.com/paper/2412.11177/) (2024)
- [Context-Guided Decompilation: A Step Towards Re-executability](https://mlsanity.com/paper/2511.01763/) (2025)
- [ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage](https://mlsanity.com/paper/2605.21615/) (2026)

## Abstract (arXiv)

Binary Code Embedding (BCE) has important applications in various reverse engineering tasks such as binary code similarity detection, type recovery, control-flow recovery and data-flow analysis. Recent studies have shown that the Transformer model can comprehend the semantics of binary code to support downstream tasks. However, existing models overlooked the prior knowledge of assembly language. In this paper, we propose a novel Transformer-based approach, namely kTrans, to generate knowledge-aware binary code embedding. By feeding explicit knowledge as additional inputs to the Transformer, and fusing implicit knowledge with a novel pre-training task, kTrans provides a new perspective to incorporating domain knowledge into a Transformer framework. We inspect the generated embeddings with outlier detection and visualization, and also apply kTrans to 3 downstream tasks: Binary Code Similarity Detection (BCSD), Function Type Recovery (FTR) and Indirect Call Recognition (ICR). Evaluation results show that kTrans can generate high-quality binary code embeddings, and outperforms state-of-the-art (SOTA) approaches on downstream tasks by 5.2%, 6.8%, and 12.6% respectively. kTrans is publicly available at: https://github.com/Learner0x5a/kTrans-release