# MERLIN -- Malware Evasion with Reinforcement LearnINg

arXiv: https://arxiv.org/abs/2203.12980 · HTML: https://mlsanity.com/paper/2203.12980/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN](https://mlsanity.com/paper/1702.05983/) (2017)
- [Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables](https://mlsanity.com/paper/1803.04173/) (2018)
- [Functionality-preserving Black-box Optimization of Adversarial Windows Malware](https://mlsanity.com/paper/2003.13526/) (2020)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [Survey of Machine Learning Techniques for Malware Analysis](https://mlsanity.com/paper/1710.08189/) (2017)
- [Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries](https://mlsanity.com/paper/1901.03583/) (2019)
- [Non-Negative Networks Against Adversarial Attacks](https://mlsanity.com/paper/1806.06108/) (2018)
- [A Survey on Adversarial Attacks for Malware Analysis](https://mlsanity.com/paper/2111.08223/) (2021)
- [Malware Analysis with Artificial Intelligence and a Particular Attention on Results Interpretability](https://mlsanity.com/paper/2107.11100/) (2021)
- [MalFox: Camouflaged Adversarial Malware Example Generation Based on Conv-GANs Against Black-Box Detectors](https://mlsanity.com/paper/2011.01509/) (2020)
- [A survey on practical adversarial examples for malware classifiers](https://mlsanity.com/paper/2011.05973/) (2020)
- [Malware Evasion Attack and Defense](https://mlsanity.com/paper/1904.05747/) (2019)

## Built on by (most cited first, 11 of 11)

- [AI-based Malware and Ransomware Detection Models](https://mlsanity.com/paper/2207.02108/) (2022)
- [Quantum Machine Learning for Malware Classification](https://mlsanity.com/paper/2305.09674/) (2023)
- [Creating Valid Adversarial Examples of Malware](https://mlsanity.com/paper/2306.13587/) (2023)
- [The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.16562/) (2023)
- [Semantic Preprocessing for LLM-based Malware Analysis](https://mlsanity.com/paper/2506.12113/) (2025)
- [Reinforcement Learning for an Efficient and Effective Malware Investigation during Cyber Incident Response](https://mlsanity.com/paper/2408.01999/) (2024)
- [EGAN: Evolutional GAN for Ransomware Evasion](https://mlsanity.com/paper/2405.12266/) (2024)
- [A Novel Reinforcement Learning Model for Post-Incident Malware Investigations](https://mlsanity.com/paper/2410.15028/) (2024)
- [Adversarial Malware Generation in Linux ELF Binaries via Semantic-Preserving Transformations](https://mlsanity.com/paper/2604.22639/) (2026)
- [Whetstones: Measuring Coevolution Between Adaptive Malware and Behavioral Defense](https://mlsanity.com/paper/2608.00745/) (2026)
- [REPLICANT: Learning Policies for Evading and Hardening Malware Detectors](https://mlsanity.com/paper/2608.28499/) (2026)

## Abstract (arXiv)

In addition to signature-based and heuristics-based detection techniques, machine learning (ML) is widely used to generalize to new, never-before-seen malicious software (malware). However, it has been demonstrated that ML models can be fooled by tricking the classifier into returning the incorrect label. These studies, for instance, usually rely on a prediction score that is fragile to gradient-based attacks. In the context of a more realistic situation where an attacker has very little information about the outputs of a malware detection engine, modest evasion rates are achieved. In this paper, we propose a method using reinforcement learning with DQN and REINFORCE algorithms to challenge two state-of-the-art ML-based detection engines (MalConv & EMBER) and a commercial AV classified by Gartner as a leader AV. Our method combines several actions, modifying a Windows portable execution (PE) file without breaking its functionalities. Our method also identifies which actions perform better and compiles a detailed vulnerability report to help mitigate the evasion. We demonstrate that REINFORCE achieves very good evasion rates even on a commercial AV with limited available information.