# Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits!

arXiv: https://arxiv.org/abs/2312.15813 · HTML: https://mlsanity.com/paper/2312.15813/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [Classifying Sequences of Extreme Length with Constant Memory Applied to Malware Detection](https://mlsanity.com/paper/2012.09390/) (2020)
- [AVClass2: Massive Malware Tag Extraction from AV Labels](https://mlsanity.com/paper/2006.10615/) (2020)
- [Automatic Yara Rule Generation Using Biclustering](https://mlsanity.com/paper/2009.03779/) (2020)
- [MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels](https://mlsanity.com/paper/2111.15031/) (2021)
- [Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints](https://mlsanity.com/paper/2108.04081/) (2021)
- [KiloGrams: Very Large N-Grams for Malware Classification](https://mlsanity.com/paper/1908.00200/) (2019)
- [Would a File by Any Other Name Seem as Malicious?](https://mlsanity.com/paper/1910.04753/) (2019)

## Built on by (most cited first, 4 of 4)

- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [Living off the Analyst: Harvesting Features from Yara Rules for Malware Detection](https://mlsanity.com/paper/2411.18516/) (2024)
- [Hamm-Grams: An Algorithm for Mining Regular Expressions of Bytes](https://mlsanity.com/paper/2607.01445/) (2026)

## Abstract (arXiv)

Industry practitioners care about small improvements in malware detection accuracy because their models are deployed to hundreds of millions of machines, meaning a 0.1% change can cause an overwhelming number of false positives. However, academic research is often restrained to public datasets on the order of ten thousand samples and is too small to detect improvements that may be relevant to industry. Working within these constraints, we devise an approach to generate a benchmark of configurable difficulty from a pool of available samples. This is done by leveraging malware family information from tools like AVClass to construct training/test splits that have different generalization rates, as measured by a secondary model. Our experiments will demonstrate that using a less accurate secondary model with disparate features is effective at producing benchmarks for a more sophisticated target model that is under evaluation. We also ablate against alternative designs to show the need for our approach.