# How to Train your Antivirus: RL-based Hardening through the Problem-Space

arXiv: https://arxiv.org/abs/2402.19027 · HTML: https://mlsanity.com/paper/2402.19027/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 72 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [Towards Robust and Reliable Algorithmic Recourse](https://mlsanity.com/paper/2102.13620/) (2021)
- [A General Framework for Adversarial Examples with Objectives](https://mlsanity.com/paper/1801.00349/) (2017)
- [Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN](https://mlsanity.com/paper/1702.05983/) (2017)
- [Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning](https://mlsanity.com/paper/1801.08917/) (2018)
- [Functionality-preserving Black-box Optimization of Adversarial Windows Malware](https://mlsanity.com/paper/2003.13526/) (2020)
- [Robust Feature-Level Adversaries are Interpretability Tools](https://mlsanity.com/paper/2110.03605/) (2021)
- [Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection](https://mlsanity.com/paper/2008.07125/) (2020)
- [On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks](https://mlsanity.com/paper/2202.03277/) (2022)
- [Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance](https://mlsanity.com/paper/2307.14657/) (2023)
- [Adversarial Examples in Constrained Domains](https://mlsanity.com/paper/2011.01183/) (2020)
- [Discriminative models for multi-instance problems with tree-structure](https://mlsanity.com/paper/1703.02868/) (2017)
- [Using Neural Network Formalism to Solve Multiple-Instance Problems](https://mlsanity.com/paper/1609.07257/) (2016)
- [A survey on practical adversarial examples for malware classifiers](https://mlsanity.com/paper/2011.05973/) (2020)
- [Realizable Universal Adversarial Perturbations for Malware](https://mlsanity.com/paper/2102.06747/) (2021)
- [Longitudinal Study of the Prevalence of Malware Evasive Techniques](https://mlsanity.com/paper/2112.11289/) (2021)
- [Improving Robustness of Malware Classifiers using Adversarial Strings Generated from Perturbed Latent Representations](https://mlsanity.com/paper/2110.11987/) (2021)
- [Explaining Classifiers Trained on Raw Hierarchical Multiple-Instance Data](https://mlsanity.com/paper/2208.02694/) (2022)

## Built on by (most cited first, 3 of 3)

- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [On the Security Risks of ML-based Malware Detection Systems: A Survey](https://mlsanity.com/paper/2505.10903/) (2025)
- [REPLICANT: Learning Policies for Evading and Hardening Malware Detectors](https://mlsanity.com/paper/2608.28499/) (2026)

## Abstract (arXiv)

ML-based malware detection on dynamic analysis reports is vulnerable to both evasion and spurious correlations. In this work, we investigate a specific ML architecture employed in the pipeline of a widely-known commercial antivirus company, with the goal to harden it against adversarial malware. Adversarial training, the sole defensive technique that can confer empirical robustness, is not applicable out of the box in this domain, for the principal reason that gradient-based perturbations rarely map back to feasible problem-space programs. We introduce a novel Reinforcement Learning approach for constructing adversarial examples, a constituent part of adversarially training a model against evasion. Our approach comes with multiple advantages. It performs modifications that are feasible in the problem-space, and only those; thus it circumvents the inverse mapping problem. It also makes possible to provide theoretical guarantees on the robustness of the model against a particular set of adversarial capabilities. Our empirical exploration validates our theoretical insights, where we can consistently reach 0% Attack Success Rate after a few adversarial retraining iterations.