# Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2402.02600 · HTML: https://mlsanity.com/paper/2402.02600/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning](https://mlsanity.com/paper/1801.08917/) (2018)
- [Functionality-preserving Black-box Optimization of Adversarial Windows Malware](https://mlsanity.com/paper/2003.13526/) (2020)
- [Exploring Adversarial Examples in Malware Detection](https://mlsanity.com/paper/1810.08280/) (2018)
- [Adversarial Attacks against Windows PE Malware Detection: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2112.12310/) (2021)
- [Binary Black-box Evasion Attacks Against Deep Learning-based Static Malware Detectors with Adversarial Byte-Level Language Model](https://mlsanity.com/paper/2012.07994/) (2020)
- [Defense Methods Against Adversarial Examples for Recurrent Neural Networks](https://mlsanity.com/paper/1901.09963/) (2019)
- [Single-Shot Black-Box Adversarial Attacks Against Malware Detectors: A Causal Language Model Approach](https://mlsanity.com/paper/2112.01724/) (2021)

## Built on by (most cited first, 1 of 1)

- [Adversarial Agents: Black-Box Evasion Attacks with Reinforcement Learning](https://mlsanity.com/paper/2503.01734/) (2025)

## Abstract (arXiv)

Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.