# Efficient Malware Detection with Optimized Learning on High-Dimensional Features

arXiv: https://arxiv.org/abs/2506.17309 · HTML: https://mlsanity.com/paper/2506.17309/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Towards an Automated Pipeline for Detecting and Classifying Malware through Machine Learning](https://mlsanity.com/paper/2106.05625/) (2021)
- [Automated Machine Learning for Deep Learning based Malware Detection](https://mlsanity.com/paper/2303.01679/) (2023)
- [Machine Learning for Detecting Malware in PE Files](https://mlsanity.com/paper/2212.13988/) (2022)

## Built on by (most cited first, 1 of 1)

- [Machine Learning Transferability for Malware Detection](https://mlsanity.com/paper/2603.26632/) (2026)

## Abstract (arXiv)

Malware detection using machine learning requires feature extraction from binary files, as models cannot process raw binaries directly. A common approach involves using LIEF for raw feature extraction and the EMBER vectorizer to generate 2381-dimensional feature vectors. However, the high dimensionality of these features introduces significant computational challenges. This study addresses these challenges by applying two dimensionality reduction techniques: XGBoost-based feature selection and Principal Component Analysis (PCA). We evaluate three reduced feature dimensions (128, 256, and 384), which correspond to approximately 5.4%, 10.8%, and 16.1% of the original 2381 features, across four models-XGBoost, LightGBM, Extra Trees, and Random Forest-using a unified training, validation, and testing split formed from the EMBER-2018, ERMDS, and BODMAS datasets. This approach ensures generalization and avoids dataset bias. Experimental results show that LightGBM trained on the 384-dimensional feature set after XGBoost feature selection achieves the highest accuracy of 97.52% on the unified dataset, providing an optimal balance between computational efficiency and detection performance. The best model, trained in 61 minutes using 30 GB of RAM and 19.5 GB of disk space, generalizes effectively to completely unseen datasets, maintaining 95.31% accuracy on TRITIUM and 93.98% accuracy on INFERNO. These findings present a scalable, compute-efficient approach for malware detection without compromising accuracy.