# On the Lack of Robustness of Binary Function Similarity Systems

arXiv: https://arxiv.org/abs/2412.04163 · HTML: https://mlsanity.com/paper/2412.04163/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [Contextualized Perturbation for Textual Adversarial Attack](https://mlsanity.com/paper/2009.07502/) (2020)
- [Discriminative Embeddings of Latent Variable Models for Structured Data](https://mlsanity.com/paper/1603.05629/) (2016)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [Functionality-preserving Black-box Optimization of Adversarial Windows Malware](https://mlsanity.com/paper/2003.13526/) (2020)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [jTrans: Jump-Aware Transformer for Binary Code Similarity](https://mlsanity.com/paper/2205.12713/) (2022)
- [Malware Makeover: Breaking ML-based Static Analysis by Modifying Executable Bytes](https://mlsanity.com/paper/1912.09064/) (2019)
- [Adversarial Music: Real World Audio Adversary Against Wake-word Detection System](https://mlsanity.com/paper/1911.00126/) (2019)
- [Challenging Machine Learning-based Clone Detectors via Semantic-preserving Code Transformations](https://mlsanity.com/paper/2111.10793/) (2021)
- [Revisiting Binary Code Similarity Analysis using Interpretable Feature Engineering and Lessons Learned](https://mlsanity.com/paper/2011.10749/) (2020)
- [Boosting Neural Networks to Decompile Optimized Binaries](https://mlsanity.com/paper/2301.00969/) (2023)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [FuncFooler: A Practical Black-box Attack Against Learning-based Binary Code Similarity Detection Methods](https://mlsanity.com/paper/2208.14191/) (2022)
- [Scalable Program Clone Search Through Spectral Analysis](https://mlsanity.com/paper/2210.13063/) (2022)
- [Adversarial Attacks against Binary Similarity Systems](https://mlsanity.com/paper/2303.11143/) (2023)

## Built on by (most cited first, 2 of 2)

- [Research on Violent Text Detection System Based on BERT-fasttext Model](https://mlsanity.com/paper/2412.16455/) (2024)
- [Walma: Learning to See Memory Corruption in WebAssembly](https://mlsanity.com/paper/2603.24167/) (2026)

## Abstract (arXiv)

Binary function similarity, which often relies on learning-based algorithms to identify what functions in a pool are most similar to a given query function, is a sought-after topic in different communities, including machine learning, software engineering, and security. Its importance stems from the impact it has in facilitating several crucial tasks, from reverse engineering and malware analysis to automated vulnerability detection. Whereas recent work cast light around performance on this long-studied problem, the research landscape remains largely lackluster in understanding the resiliency of the state-of-the-art machine learning models against adversarial attacks. As security requires to reason about adversaries, in this work we assess the robustness of such models through a simple yet effective black-box greedy attack, which modifies the topology and the content of the control flow of the attacked functions. We demonstrate that this attack is successful in compromising all the models, achieving average attack success rates of 57.06% and 95.81% depending on the problem settings (targeted and untargeted attacks). Our findings are insightful: top performance on clean data does not necessarily relate to top robustness properties, which explicitly highlights performance-robustness trade-offs one should consider when deploying such models, calling for further research.