# A Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer

arXiv: https://arxiv.org/abs/2412.11177 · HTML: https://mlsanity.com/paper/2412.11177/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [Microsoft Malware Classification Challenge](https://mlsanity.com/paper/1802.10135/) (2018)
- [jTrans: Jump-Aware Transformer for Binary Code Similarity](https://mlsanity.com/paper/2205.12713/) (2022)
- [Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity](https://mlsanity.com/paper/2012.08680/) (2020)
- [SAFE: Self-Attentive Function Embeddings for Binary Similarity](https://mlsanity.com/paper/1811.05296/) (2018)
- [Classifying Sequences of Extreme Length with Constant Memory Applied to Malware Detection](https://mlsanity.com/paper/2012.09390/) (2020)
- [XDA: Accurate, Robust Disassembly with Transfer Learning](https://mlsanity.com/paper/2010.00770/) (2020)
- [Revisiting Binary Code Similarity Analysis using Interpretable Feature Engineering and Lessons Learned](https://mlsanity.com/paper/2011.10749/) (2020)
- [CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code](https://mlsanity.com/paper/2310.16853/) (2023)
- [kTrans: Knowledge-Aware Transformer for Binary Code Embedding](https://mlsanity.com/paper/2308.12659/) (2023)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [Asteria-Pro: Enhancing Deep-Learning Based Binary Code Similarity Detection by Incorporating Domain Knowledge](https://mlsanity.com/paper/2301.00511/) (2023)
- [Callee: Recovering Call Graphs for Binaries with Transfer and Contrastive Learning](https://mlsanity.com/paper/2111.01415/) (2021)
- [NeuDep: Neural Binary Memory Dependence Analysis](https://mlsanity.com/paper/2210.02853/) (2022)
- [o-glasses: Visualizing x86 Code from Binary Using a 1d-CNN](https://mlsanity.com/paper/1806.05328/) (2018)
- [Machine Learning for Windows Malware Detection and Classification: Methods, Challenges and Ongoing Research](https://mlsanity.com/paper/2404.18541/) (2024)

## Built on by (most cited first, 1 of 1)

- [Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis](https://mlsanity.com/paper/2511.17573/) (2025)

## Abstract (arXiv)

Language model approaches have recently been integrated into binary analysis tasks, such as function similarity detection and function signature recovery. These models typically employ a two-stage training process: pre-training via Masked Language Modeling (MLM) on machine code and fine-tuning for specific tasks. While MLM helps to understand binary code structures, it ignores essential code characteristics, including control and data flow, which negatively affect model generalization. Recent work leverages domain-specific features (e.g., control flow graphs and dynamic execution traces) in transformer-based approaches to improve binary code semantic understanding. However, this approach involves complex feature engineering, a cumbersome and time-consuming process that can introduce predictive uncertainty when dealing with stripped or obfuscated code, leading to a performance drop. In this paper, we introduce ProTST, a novel transformer-based methodology for binary code embedding. ProTST employs a hierarchical training process based on a unique tree-like structure, where knowledge progressively flows from fundamental tasks at the root to more specialized tasks at the leaves. This progressive teacher-student paradigm allows the model to build upon previously learned knowledge, resulting in high-quality embeddings that can be effectively leveraged for diverse downstream binary analysis tasks. The effectiveness of ProTST is evaluated in seven binary analysis tasks, and the results show that ProTST yields an average validation score (F1, MRR, and Recall@1) improvement of 14.8% compared to traditional two-stage training and an average validation score of 10.7% compared to multimodal two-stage frameworks.