# Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries

arXiv: https://arxiv.org/abs/2301.01701 · HTML: https://mlsanity.com/paper/2301.01701/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 4 Jan 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021)
- [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019)
- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [DIRE: A Neural Approach to Decompiled Identifier Naming](https://mlsanity.com/paper/1909.09029/) (2019)
- [Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modeling](https://mlsanity.com/paper/2103.12801/) (2021)
- [Learning to Find Usages of Library Functions in Optimized Binaries](https://mlsanity.com/paper/2103.05221/) (2021)

## Built on by (most cited first, 26 of 26)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Language Models for Code Completion: A Practical Evaluation](https://mlsanity.com/paper/2402.16197/) (2024)
- [Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models](https://mlsanity.com/paper/2312.09601/) (2023)
- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [The (ab)use of Open Source Code to Train Large Language Models](https://mlsanity.com/paper/2302.13681/) (2023)
- [Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge](https://mlsanity.com/paper/2302.07735/) (2023)
- [Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study](https://mlsanity.com/paper/2304.12269/) (2023)
- [Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases](https://mlsanity.com/paper/2405.19581/) (2024)
- [Exploring the Efficacy of Large Language Models (GPT-4) in Binary Reverse Engineering](https://mlsanity.com/paper/2406.06637/) (2024)
- [Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning](https://mlsanity.com/paper/2509.13755/) (2025)
- [Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks](https://mlsanity.com/paper/2504.01850/) (2025)
- [STACC: Code Comment Classification using SentenceTransformers](https://mlsanity.com/paper/2302.13149/) (2023)
- [MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization](https://mlsanity.com/paper/2406.18379/) (2024)
- [On the Impact of Language Selection for Training and Evaluating Programming Language Models](https://mlsanity.com/paper/2308.13354/) (2023)
- [ASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuning](https://mlsanity.com/paper/2503.11617/) (2025)
- [How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning](https://mlsanity.com/paper/2501.17501/) (2025)
- [LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine](https://mlsanity.com/paper/2404.03664/) (2024)
- [REx86: A Local Large Language Model for Assisting in x86 Assembly Reverse Engineering](https://mlsanity.com/paper/2510.20975/) (2025)
- [Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysis](https://mlsanity.com/paper/2511.11439/) (2025)
- [C-ing Clearly: Enhanced Binary Code Explanations using C code](https://mlsanity.com/paper/2512.14500/) (2025)
- [SoK: AI-Augmented Binary Reversing](https://mlsanity.com/paper/2606.17398/) (2026)
- [OASIF: An Efficient Obfuscation-Aware Self-Improving Framework for LLM-Based Assembly Code Instruction Following and Comprehension](https://mlsanity.com/paper/2606.29155/) (2026)
- [Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering](https://mlsanity.com/paper/2608.07038/) (2026)
- [Reveree: Diagnosing LLM Reverse-Engineering Agents](https://mlsanity.com/paper/2609.01185/) (2026)

## Abstract (arXiv)

Reverse engineering binaries is required to understand and analyse programs for which the source code is unavailable. Decompilers can transform the largely unreadable binaries into a more readable source code-like representation. However, reverse engineering is time-consuming, much of which is taken up by labelling the functions with semantic information. While the automated summarisation of decompiled code can help Reverse Engineers understand and analyse binaries, current work mainly focuses on summarising source code, and no suitable dataset exists for this task. In this work, we extend large pre-trained language models of source code to summarise decompiled binary functions. Furthermore, we investigate the impact of input and data properties on the performance of such models. Our approach consists of two main components; the data and the model. We first build CAPYBARA, a dataset of 214K decompiled function-documentation pairs across various compiler optimisations. We extend CAPYBARA further by generating synthetic datasets and deduplicating the data. Next, we fine-tune the CodeT5 base model with CAPYBARA to create BinT5. BinT5 achieves the state-of-the-art BLEU-4 score of 60.83, 58.82, and 44.21 for summarising source, decompiled, and synthetically stripped decompiled code, respectively. This indicates that these models can be extended to decompiled binaries successfully. Finally, we found that the performance of BinT5 is not heavily dependent on the dataset size and compiler optimisation level. We recommend future research to further investigate transferring knowledge when working with less expressive input formats such as stripped binaries.