# EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability

arXiv: https://arxiv.org/abs/2607.24177 · HTML: https://mlsanity.com/paper/2607.24177/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [(De)Randomized Smoothing for Certifiable Defense against Patch Attacks](https://mlsanity.com/paper/2002.10733/) (2020)
- [Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables](https://mlsanity.com/paper/1803.04173/) (2018)
- [Functionality-preserving Black-box Optimization of Adversarial Windows Malware](https://mlsanity.com/paper/2003.13526/) (2020)
- [Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints](https://mlsanity.com/paper/2102.12827/) (2021)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features](https://mlsanity.com/paper/1508.03096/) (2015)
- [Malware Makeover: Breaking ML-based Static Analysis by Modifying Executable Bytes](https://mlsanity.com/paper/1912.09064/) (2019)
- [Novel Feature Extraction, Selection and Fusion for Effective Malware Family Classification](https://mlsanity.com/paper/1511.04317/) (2015)
- [EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers](https://mlsanity.com/paper/2506.05074/) (2025)
- [RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletion](https://mlsanity.com/paper/2302.01757/) (2023)
- [Activation Analysis of a Byte-Based Deep Neural Network for Malware Classification](https://mlsanity.com/paper/1903.04717/) (2019)
- [AttackBench: Evaluating Gradient-based Attacks for Adversarial Examples](https://mlsanity.com/paper/2404.19460/) (2024)
- [Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations](https://mlsanity.com/paper/2208.12248/) (2022)
- [Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing](https://mlsanity.com/paper/2308.08906/) (2023)
- [A Comparison of Adversarial Learning Techniques for Malware Detection](https://mlsanity.com/paper/2308.09958/) (2023)
- [A Robust Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via (De)Randomized Smoothing](https://mlsanity.com/paper/2402.15267/) (2024)
- [DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness](https://mlsanity.com/paper/2303.13372/) (2023)
- [Optimized Approaches to Malware Detection: A Study of Machine Learning and Deep Learning Techniques](https://mlsanity.com/paper/2504.17930/) (2025)
- [Updating Windows Malware Detectors: Balancing Robustness and Regression against Adversarial EXEmples](https://mlsanity.com/paper/2405.02646/) (2024)

## Abstract (arXiv)

Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.