# Adaptive Malware Detection using Sequential Feature Selection: A Dueling Double Deep Q-Network (D3QN) Framework for Intelligent Classification

arXiv: https://arxiv.org/abs/2507.04372 · HTML: https://mlsanity.com/paper/2507.04372/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 6 Jul 2025.

## Builds on (most cited first, 7 of 7)

- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Microsoft Malware Classification Challenge](https://mlsanity.com/paper/1802.10135/) (2018)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [Novel Feature Extraction, Selection and Fusion for Effective Malware Family Classification](https://mlsanity.com/paper/1511.04317/) (2015)
- [Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems](https://mlsanity.com/paper/2410.24017/) (2024)
- [DroidRL: Reinforcement Learning Driven Feature Selection for Android Malware Detection](https://mlsanity.com/paper/2203.02719/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution](https://mlsanity.com/paper/2606.30572/) (2026)

## Abstract (arXiv)

Traditional malware detection methods exhibit computational inefficiency due to exhaustive feature extraction requirements, creating accuracy-efficiency trade-offs that limit real-time deployment. We formulate malware classification as a Markov Decision Process with episodic feature acquisition and propose a Dueling Double Deep Q-Network (D3QN) framework for adaptive sequential feature selection. The agent learns to dynamically select informative features per sample before terminating with classification decisions, optimizing both detection accuracy and computational cost through reinforcement learning. We evaluate our approach on Microsoft Big2015 (9-class, 1,795 features) and BODMAS (binary, 2,381 features) datasets. D3QN achieves 99.22% and 98.83% accuracy while utilizing only 61 and 56 features on average, representing 96.6% and 97.6% dimensionality reduction. This yields computational efficiency improvements of 30.1x and 42.5x over traditional ensemble methods. Comprehensive ablation studies demonstrate consistent superiority over Random Forest, XGBoost, and static feature selection approaches. Quantitative analysis demonstrates that D3QN learns non-random feature selection policies with 62.5% deviation from uniform baseline distributions. The learned policies exhibit structured hierarchical preferences, utilizing high-level metadata features for initial assessment while selectively incorporating detailed behavioral features based on classification uncertainty. Feature specialization analysis reveals 57.7% of examined features demonstrate significant class-specific discrimination patterns. Our results validate reinforcement learning-based sequential feature selection for malware classification, achieving superior accuracy with substantial computational reduction through learned adaptive policies.