# Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance

arXiv: https://arxiv.org/abs/2307.14657 · HTML: https://mlsanity.com/paper/2307.14657/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 64 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Drift](https://mlsanity.com/paper/2010.03856/) (2020)
- [Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features](https://mlsanity.com/paper/1508.03096/) (2015)
- [Dynamic Malware Analysis with Feature Engineering and Feature Learning](https://mlsanity.com/paper/1907.07352/) (2019)
- [Reviewer Integration and Performance Measurement for Malware Detection](https://mlsanity.com/paper/1510.07338/) (2015)
- [AVClass2: Massive Malware Tag Extraction from AV Labels](https://mlsanity.com/paper/2006.10615/) (2020)
- [MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels](https://mlsanity.com/paper/2111.15031/) (2021)
- [Early Stage Malware Prediction Using Recurrent Neural Networks](https://mlsanity.com/paper/1708.03513/) (2017)
- [Neurlux: Dynamic Malware Analysis Without Feature Engineering](https://mlsanity.com/paper/1910.11376/) (2019)
- [Towards an Automated Pipeline for Detecting and Classifying Malware through Machine Learning](https://mlsanity.com/paper/2106.05625/) (2021)
- [Portable, Data-Driven Malware Detection using Language Processing and Machine Learning Techniques on Behavioral Analysis Reports](https://mlsanity.com/paper/1812.10327/) (2018)
- [Longitudinal Study of the Prevalence of Malware Evasive Techniques](https://mlsanity.com/paper/2112.11289/) (2021)
- [Challenges in Forecasting Malicious Events from Incomplete Data](https://mlsanity.com/paper/2004.04597/) (2020)

## Built on by (most cited first, 13 of 13)

- [Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems](https://mlsanity.com/paper/2410.24017/) (2024)
- [SLIFER: Investigating Performance and Robustness of Malware Detection Pipelines](https://mlsanity.com/paper/2405.14478/) (2024)
- [How to Train your Antivirus: RL-based Hardening through the Problem-Space](https://mlsanity.com/paper/2402.19027/) (2024)
- [MalMixer: Few-Shot Malware Classification with Retrieval-Augmented Semi-Supervised Learning](https://mlsanity.com/paper/2409.13213/) (2024)
- [Trust Under Siege: Label Spoofing Attacks against Machine Learning for Android Malware Detection](https://mlsanity.com/paper/2503.11841/) (2025)
- [On the Security Risks of ML-based Malware Detection Systems: A Survey](https://mlsanity.com/paper/2505.10903/) (2025)
- [ADAPT: A Pseudo-labeling Approach to Combat Concept Drift in Malware Detection](https://mlsanity.com/paper/2507.08597/) (2025)
- [AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research](https://mlsanity.com/paper/2511.13333/) (2025)
- [Rethinking and Exploring String-Based Malware Family Classification in the Era of LLMs and RAG](https://mlsanity.com/paper/2507.04055/) (2025)
- [Certifiably robust malware detectors by design](https://mlsanity.com/paper/2508.10038/) (2025)
- [LLM-Driven Feature-Level Adversarial Attacks on Android Malware Detectors](https://mlsanity.com/paper/2512.21404/) (2025)
- [Attacking the First-Principle: A Black-Box, Query-Free Targeted Mimicry Attack on Binary Function Classifiers](https://mlsanity.com/paper/2605.18231/) (2026)
- [Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineering](https://mlsanity.com/paper/2610.14960/) (2026)

## Abstract (arXiv)

Many studies have proposed machine-learning (ML) models for malware detection and classification, reporting an almost-perfect performance. However, they assemble ground-truth in different ways, use diverse static- and dynamic-analysis techniques for feature extraction, and even differ on what they consider a malware family. As a consequence, our community still lacks an understanding of malware classification results: whether they are tied to the nature and distribution of the collected dataset, to what extent the number of families and samples in the training dataset influence performance, and how well static and dynamic features complement each other. This work sheds light on those open questions. by investigating the key factors influencing ML-based malware detection and classification. For this, we collect the largest balanced malware dataset so far with 67K samples from 670 families (100 samples each), and train state-of-the-art models for malware detection and family classification using our dataset. Our results reveal that static features perform better than dynamic features, and that combining both only provides marginal improvement over static features. We discover no correlation between packing and classification accuracy, and that missing behaviors in dynamically-extracted features highly penalize their performance. We also demonstrate how a larger number of families to classify make the classification harder, while a higher number of samples per family increases accuracy. Finally, we find that models trained on a uniform distribution of samples per family better generalize on unseen data.