# DRMD: Deep Reinforcement Learning for Malware Detection under Concept Drift

arXiv: https://arxiv.org/abs/2508.18839 · HTML: https://mlsanity.com/paper/2508.18839/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Drift](https://mlsanity.com/paper/2010.03856/) (2020)
- [Classification with Costly Features using Deep Reinforcement Learning](https://mlsanity.com/paper/1711.07364/) (2017)
- [Deep reinforcement learning in medical imaging: A literature review](https://mlsanity.com/paper/2103.05115/) (2021)
- [Continuous Learning for Android Malware Detection](https://mlsanity.com/paper/2302.04332/) (2023)
- [Deep Reinforcement Learning for Imbalanced Classification](https://mlsanity.com/paper/1901.01379/) (2019)
- [Reviewer Integration and Performance Measurement for Malware Detection](https://mlsanity.com/paper/1510.07338/) (2015)
- [Reward Shaping for Happier Autonomous Cyber Security Agents](https://mlsanity.com/paper/2310.13565/) (2023)
- [Inroads into Autonomous Network Defence using Explained Reinforcement Learning](https://mlsanity.com/paper/2306.09318/) (2023)
- [Autonomous Network Defence using Reinforcement Learning](https://mlsanity.com/paper/2409.18197/) (2024)
- [Neural Active Learning Beyond Bandits](https://mlsanity.com/paper/2404.12522/) (2024)
- [Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning](https://mlsanity.com/paper/2312.04940/) (2023)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)](https://mlsanity.com/paper/2402.01359/) (2024)
- [Falcon: Fair Active Learning using Multi-armed Bandits](https://mlsanity.com/paper/2401.12722/) (2024)
- [Beyond the TESSERACT:Trustworthy Dataset Curation for Sound Evaluations of Android Malware Classifiers](https://mlsanity.com/paper/2506.23814/) (2025)
- [APIRL: Deep Reinforcement Learning for REST API Fuzzing](https://mlsanity.com/paper/2412.15991/) (2024)
- [Deep Reinforcement Learning for Multi-class Imbalanced Training](https://mlsanity.com/paper/2205.12070/) (2022)

## Built on by (most cited first, 7 of 7)

- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning](https://mlsanity.com/paper/2604.15705/) (2026)
- [Trident: Improving Malware Detection with LLMs and Behavioral Features](https://mlsanity.com/paper/2605.00297/) (2026)
- [Clutch Control: An Attention-based Combinatorial Bandit for Efficient Mutation in JavaScript Engine Fuzzing](https://mlsanity.com/paper/2510.12732/) (2025)
- [Concept Drift Adaptation Using Self-Supervised and Reinforcement Learning In Android Malware Detection](https://mlsanity.com/paper/2605.24294/) (2026)
- [REPLICANT: Learning Policies for Evading and Hardening Malware Detectors](https://mlsanity.com/paper/2608.28499/) (2026)

## Abstract (arXiv)

Malware detection in real-world settings must deal with evolving threats, limited labeling budgets, and uncertain predictions. Traditional classifiers, without additional mechanisms, struggle to maintain performance under concept drift in malware domains, as their supervised learning formulation cannot optimize when to defer decisions to manual labeling and adaptation. Modern malware detection pipelines combine classifiers with monthly active learning (AL) and rejection mechanisms to mitigate the impact of concept drift. In this work, we develop a novel formulation of malware detection as a one-step Markov Decision Process and train a deep reinforcement learning (DRL) agent, simultaneously optimizing sample classification performance and rejecting high-risk samples for manual labeling. We evaluated the joint detection and drift mitigation policy learned by the DRL-based Malware Detection (DRMD) agent through time-aware evaluations on Android malware datasets subject to realistic drift requiring multi-year performance stability. The policies learned under these conditions achieve a higher Area Under Time (AUT) performance compared to standard classification approaches used in the domain, showing improved resilience to concept drift. Specifically, the DRMD agent achieved an average AUT improvement of 8.66 and 10.90 for the classification-only and classification-rejection policies, respectively. Our results demonstrate for the first time that DRL can facilitate effective malware detection and improved resiliency to concept drift in the dynamic setting of Android malware detection.