EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability
Not on the map. Why?
Abstract (arXiv)
Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
27 of 50 references.
References · 27
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
- 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 414
- 2018EMBER: An Open Dataset for Training Static PE Malware Machine Learning Modelscited by 165
- 2017Malware Detection by Eating a Whole EXEcited by 115
- 2018TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Timecited by 108
- 2020(De)Randomized Smoothing for Certifiable Defense against Patch Attackscited by 98
- 2018Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executablescited by 62
- 2020Functionality-preserving Black-box Optimization of Adversarial Windows Malwarecited by 58
- 2021Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraintscited by 50
15 more
- 2020SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detectioncited by 49
- 2015Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Featurescited by 48
- 2019Malware Makeover: Breaking ML-based Static Analysis by Modifying Executable Bytescited by 31
- 2015Novel Feature Extraction, Selection and Fusion for Effective Malware Family Classificationcited by 21
- 2025EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifierscited by 19
- 2023RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletioncited by 17
- 2019Activation Analysis of a Byte-Based Deep Neural Network for Malware Classificationcited by 14
- 2024AttackBench: Evaluating Gradient-based Attacks for Adversarial Examplescited by 14
- 2022Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representationscited by 9
- 2023Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothingcited by 5
- 2023A Comparison of Adversarial Learning Techniques for Malware Detectioncited by 5
- 2024A Robust Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via (De)Randomized Smoothingcited by 4
- 2023DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustnesscited by 3
- 2025Optimized Approaches to Malware Detection: A Study of Machine Learning and Deep Learning Techniquescited by 2
- 2024Updating Windows Malware Detectors: Balancing Robustness and Regression against Adversarial EXEmplescited by 1
Built on by
Not cited yet.