# Pre-Training Representations of Binary Code Using Contrastive Learning arXiv: https://arxiv.org/abs/2210.05102 ยท HTML: https://mlsanity.com/paper/2210.05102/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 60 references that are in the map. - First posted on arXiv 11 Oct 2022. ## Builds on (most cited first, 30 of 33) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020) - [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020) - [A Survey on Multi-view Learning](https://mlsanity.com/paper/1304.5634/) (2013) - [SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities](https://mlsanity.com/paper/1807.06756/) (2018) - [Imbalance Trouble: Revisiting Neural-Collapse Geometry](https://mlsanity.com/paper/2208.05512/) (2022) - [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020) - [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021) - [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018) - [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021) - [DIRE: A Neural Approach to Decompiled Identifier Naming](https://mlsanity.com/paper/1909.09029/) (2019) - [CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2104.02443/) (2021) - [Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs](https://mlsanity.com/paper/1808.04706/) (2018) - [Augmenting Decompiler Output with Learned Variable Names and Types](https://mlsanity.com/paper/2108.06363/) (2021) - [A Survey of Binary Code Similarity](https://mlsanity.com/paper/1909.11424/) (2019) - [Learning Execution through Neural Code Fusion](https://mlsanity.com/paper/1906.07181/) (2019) - [Bin2vec: Learning Representations of Binary Executable Programs for Security Tasks](https://mlsanity.com/paper/2002.03388/) (2020) ## Built on by (most cited first, 6 of 6) - [kTrans: Knowledge-Aware Transformer for Binary Code Embedding](https://mlsanity.com/paper/2308.12659/) (2023) - [Source Code Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2305.19915/) (2023) - [BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding](https://mlsanity.com/paper/2409.07889/) (2024) - [Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.16692/) (2025) - [Identifier-Free Code Embedding Models for Scalable Search](https://mlsanity.com/paper/2605.05251/) (2026) - [SoK: AI-Augmented Binary Reversing](https://mlsanity.com/paper/2606.17398/) (2026) ## Abstract (arXiv) Binary code analysis and comprehension is critical to applications in reverse engineering and computer security tasks where source code is not available. Unfortunately, unlike source code, binary code lacks semantics and is more difficult for human engineers to understand and analyze. In this paper, we present ContraBin, a contrastive learning technique that integrates source code and comment information along with binaries to create an embedding capable of aiding binary analysis and comprehension tasks. Specifically, we present three components in ContraBin: (1) a primary contrastive learning method for initial pre-training, (2) a simplex interpolation method to integrate source code, comments, and binary code, and (3) an intermediate representation learning algorithm to train a binary code embedding. We further analyze the impact of human-written and synthetic comments on binary code comprehension tasks, revealing a significant performance disparity. While synthetic comments provide substantial benefits, human-written comments are found to introduce noise, even resulting in performance drops compared to using no comments. These findings reshape the narrative around the role of comment types in binary code analysis. We evaluate the effectiveness of ContraBin through four indicative downstream tasks related to binary code: algorithmic functionality classification, function name recovery, code summarization, and reverse engineering. The results show that ContraBin considerably improves performance on all four tasks, measured by accuracy, mean of average precision, and BLEU scores as appropriate. ContraBin is the first language representation model to incorporate source code, binary code, and comments into contrastive code representation learning and is intended to contribute to the field of binary code analysis. The dataset used in this study is available for further research.