# Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

arXiv: https://arxiv.org/abs/1801.08917 · HTML: https://mlsanity.com/paper/1801.08917/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2018.

## Built on by (most cited first, 30 of 61)

- [The Threat of Offensive AI to Organizations](https://mlsanity.com/paper/2106.15764/) (2021)
- [Adversarial Attacks against Windows PE Malware Detection: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2112.12310/) (2021)
- [On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks](https://mlsanity.com/paper/2202.03277/) (2022)
- [Jigsaw Puzzle: Selective Backdoor Attack to Subvert Malware Classifiers](https://mlsanity.com/paper/2202.05470/) (2022)
- [Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference](https://mlsanity.com/paper/2108.11673/) (2021)
- [EvilModel 2.0: Bringing Neural Network Models into Malware Attacks](https://mlsanity.com/paper/2109.04344/) (2021)
- [EvadeDroid: A Practical Evasion Attack on Machine Learning for Black-box Android Malware Detection](https://mlsanity.com/paper/2110.03301/) (2021)
- [Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware](https://mlsanity.com/paper/2607.02357/) (2026)
- [Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations](https://mlsanity.com/paper/2208.12248/) (2022)
- [MORPH: Towards Automated Concept Drift Adaptation for Malware Detection](https://mlsanity.com/paper/2401.12790/) (2024)
- [RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation](https://mlsanity.com/paper/2405.03064/) (2024)
- [Explanation-Guided Diagnosis of Machine Learning Evasion Attacks](https://mlsanity.com/paper/2106.15820/) (2021)
- [Fast & Furious: Modelling Malware Detection as Evolving Data Streams](https://mlsanity.com/paper/2205.12311/) (2022)
- [Efficient Malware Analysis Using Metric Embeddings](https://mlsanity.com/paper/2212.02663/) (2022)
- [Realizable Universal Adversarial Perturbations for Malware](https://mlsanity.com/paper/2102.06747/) (2021)
- [Automating Privilege Escalation with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.01362/) (2021)
- [A Comparison of Adversarial Learning Techniques for Malware Detection](https://mlsanity.com/paper/2308.09958/) (2023)
- [The Adaptive Arms Race: Redefining Robustness in AI Security](https://mlsanity.com/paper/2312.13435/) (2023)
- [SoK: On the Offensive Potential of AI](https://mlsanity.com/paper/2412.18442/) (2024)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [Single-Shot Black-Box Adversarial Attacks Against Malware Detectors: A Causal Language Model Approach](https://mlsanity.com/paper/2112.01724/) (2021)
- [Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations](https://mlsanity.com/paper/2512.22883/) (2025)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [Mal2GCN: A Robust Malware Detection Approach Using Deep Graph Convolutional Networks With Non-Negative Weights](https://mlsanity.com/paper/2108.12473/) (2021)
- [StratDef: Strategic Defense Against Adversarial Attacks in ML-based Malware Detection](https://mlsanity.com/paper/2202.07568/) (2022)
- [Deep Reinforcement Learning based Evasion Generative Adversarial Network for Botnet Detection](https://mlsanity.com/paper/2210.02840/) (2022)
- [Quantum Machine Learning for Malware Classification](https://mlsanity.com/paper/2305.09674/) (2023)
- [Creating Valid Adversarial Examples of Malware](https://mlsanity.com/paper/2306.13587/) (2023)
- [URET: Universal Robustness Evaluation Toolkit (for Evasion)](https://mlsanity.com/paper/2308.01840/) (2023)
- [The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.16562/) (2023)

## Abstract (arXiv)

Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. The general framework does not require a differentiable model nor does it require the engine to produce a score. Instead, an RL agent is equipped with a set of functionality-preserving operations that it may perform on the PE file. Through a series of games played against the anti-malware engine, it learns which sequences of operations are likely to result in evading the detector for any given malware sample. This enables completely black-box attacks against static PE anti-malware, and produces functional evasive malware samples as a direct result. We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset. We demonstrate that attacks against this model appear to also evade components of publicly hosted antivirus engines. Adversarial training results are also presented: by retraining the model on evasive ransomware samples, a subsequent attack is 33% less effective. However, there are overfitting dangers when adversarial training, which we note. We release code to allow researchers to reproduce and improve this approach.