# PalmTree: Learning an Assembly Language Model for Instruction Embedding

arXiv: https://arxiv.org/abs/2103.03809 · HTML: https://mlsanity.com/paper/2103.03809/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [Discriminative Embeddings of Latent Variable Models for Structured Data](https://mlsanity.com/paper/1603.05629/) (2016)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [Automated software vulnerability detection with machine learning](https://mlsanity.com/paper/1803.04497/) (2018)
- [Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity](https://mlsanity.com/paper/2012.08680/) (2020)
- [SAFE: Self-Attentive Function Embeddings for Binary Similarity](https://mlsanity.com/paper/1811.05296/) (2018)
- [A Survey of Word Embeddings Evaluation Methods](https://mlsanity.com/paper/1801.09536/) (2018)
- [Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs](https://mlsanity.com/paper/1808.04706/) (2018)

## Built on by (most cited first, 30 of 49)

- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision](https://mlsanity.com/paper/2402.16928/) (2024)
- [Beyond the C: Retargetable Decompilation using Neural Machine Translation](https://mlsanity.com/paper/2212.08950/) (2022)
- [Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models](https://mlsanity.com/paper/2312.09601/) (2023)
- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [Exploiting Code Symmetries for Learning Program Semantics](https://mlsanity.com/paper/2308.03312/) (2023)
- [kTrans: Knowledge-Aware Transformer for Binary Code Embedding](https://mlsanity.com/paper/2308.12659/) (2023)
- [SOS! Soft Prompt Attack Against Open-Source Large Language Models](https://mlsanity.com/paper/2407.03160/) (2024)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code](https://mlsanity.com/paper/2601.20679/) (2026)
- [Decompiling x86 Deep Neural Network Executables](https://mlsanity.com/paper/2210.01075/) (2022)
- [Efficient Malware Analysis Using Metric Embeddings](https://mlsanity.com/paper/2212.02663/) (2022)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [XFL: Naming Functions in Binaries with Extreme Multi-label Learning](https://mlsanity.com/paper/2107.13404/) (2021)
- [MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization](https://mlsanity.com/paper/2406.18379/) (2024)
- [Callee: Recovering Call Graphs for Binaries with Transfer and Contrastive Learning](https://mlsanity.com/paper/2111.01415/) (2021)
- [FASER: Binary Code Similarity Search through the use of Intermediate Representations](https://mlsanity.com/paper/2310.03605/) (2023)
- [Unraveling the Key of Machine Learning-based Android Malware Detection](https://mlsanity.com/paper/2402.02953/) (2024)
- [NeuDep: Neural Binary Memory Dependence Analysis](https://mlsanity.com/paper/2210.02853/) (2022)
- [UniASM: Binary Code Similarity Detection without Fine-tuning](https://mlsanity.com/paper/2211.01144/) (2022)
- [BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding](https://mlsanity.com/paper/2409.07889/) (2024)
- [How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis](https://mlsanity.com/paper/2511.03825/) (2025)
- [Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning](https://mlsanity.com/paper/2301.08427/) (2023)
- [Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code](https://mlsanity.com/paper/2405.20611/) (2024)
- [ViC: Virtual Compiler Is All You Need For Assembly Code Search](https://mlsanity.com/paper/2408.06385/) (2024)
- [On the Lack of Robustness of Binary Function Similarity Systems](https://mlsanity.com/paper/2412.04163/) (2024)
- [Tady: A Neural Disassembler without Structural Constraint Violations](https://mlsanity.com/paper/2506.13323/) (2025)
- [VEXIR2Vec: An Architecture-Neutral Embedding Framework for Binary Similarity](https://mlsanity.com/paper/2312.00507/) (2023)
- [Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code](https://mlsanity.com/paper/2409.17513/) (2024)

## Abstract (arXiv)

Deep learning has demonstrated its strengths in numerous binary analysis tasks, including function boundary detection, binary code search, function prototype inference, value set analysis, etc. When applying deep learning to binary analysis tasks, we need to decide what input should be fed into the neural network model. More specifically, we need to answer how to represent an instruction in a fixed-length vector. The idea of automatically learning instruction representations is intriguing, however the existing schemes fail to capture the unique characteristics of disassembly. These schemes ignore the complex intra-instruction structures and mainly rely on control flow in which the contextual information is noisy and can be influenced by compiler optimizations. In this paper, we propose to pre-train an assembly language model called PalmTree for generating general-purpose instruction embeddings by conducting self-supervised training on large-scale unlabeled binary corpora. PalmTree utilizes three pre-training tasks to capture various characteristics of assembly language. These training tasks overcome the problems in existing schemes, thus can help to generate high-quality representations. We conduct both intrinsic and extrinsic evaluations, and compare PalmTree with other instruction embedding schemes. PalmTree has the best performance for intrinsic metrics, and outperforms the other instruction embedding schemes for all downstream tasks.