# On the impact of dataset size and class imbalance in evaluating machine-learning-based windows malware detection techniques

arXiv: https://arxiv.org/abs/2206.06256 · HTML: https://mlsanity.com/paper/2206.06256/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 75 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 6 of 6)

- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Survey of Machine Learning Techniques for Malware Analysis](https://mlsanity.com/paper/1710.08189/) (2017)
- [Ransomware Payments in the Bitcoin Ecosystem](https://mlsanity.com/paper/1804.04080/) (2018)
- [Getting Passive Aggressive About False Positives: Patching Deployed Malware Detectors](https://mlsanity.com/paper/2010.12080/) (2020)
- [Malware Detection and Analysis: Challenges and Research Opportunities](https://mlsanity.com/paper/2101.08429/) (2021)

## Abstract (arXiv)

The purpose of this project was to collect and analyse data about the comparability and real-life applicability of published results focusing on Microsoft Windows malware, more specifically the impact of dataset size and testing dataset imbalance on measured detector performance. Some researchers use smaller datasets, and if dataset size has a significant impact on performance, that makes comparison of the published results difficult. Researchers also tend to use balanced datasets and accuracy as a metric for testing. The former is not a true representation of reality, where benign samples significantly outnumber malware, and the latter is approach is known to be problematic for imbalanced problems. The project identified two key objectives, to understand if dataset size correlates to measured detector performance to an extent that prevents meaningful comparison of published results, and to understand if good performance reported in published research can be expected to perform well in a real-world deployment scenario. The research's results suggested that dataset size does correlate with measured detector performance to an extent that prevents meaningful comparison of published results, and without understanding the nature of the training set size-accuracy curve for published results conclusions between approaches on which approach is "better" shouldn't be made solely based on accuracy scores. Results also suggested that high accuracy scores don't necessarily translate to high real-world performance.