# Adversarial Co-Evolution of Malware and Detection Models: A Bilevel Optimization Perspective

arXiv: https://arxiv.org/abs/2604.22569 · HTML: https://mlsanity.com/paper/2604.22569/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Exploring Adversarial Examples in Malware Detection](https://mlsanity.com/paper/1810.08280/) (2018)
- [A Survey on Adversarial Attacks for Malware Analysis](https://mlsanity.com/paper/2111.08223/) (2021)
- [MAB-Malware: A Reinforcement Learning Framework for Attacking Static Malware Classifiers](https://mlsanity.com/paper/2003.03100/) (2020)
- [A Comparison of Adversarial Learning Techniques for Malware Detection](https://mlsanity.com/paper/2308.09958/) (2023)
- [A Game-Theoretic Analysis of Adversarial Classification](https://mlsanity.com/paper/1610.04972/) (2016)
- [RawMal-TF: Raw Malware Dataset Labeled by Type and Family](https://mlsanity.com/paper/2506.23909/) (2025)
- [Creating Valid Adversarial Examples of Malware](https://mlsanity.com/paper/2306.13587/) (2023)
- [Combining Generators of Adversarial Malware Examples to Increase Evasion Rate](https://mlsanity.com/paper/2304.07360/) (2023)
- [Detecting and Explaining Malware Family Evolution Using Rule-Based Drift Analysis](https://mlsanity.com/paper/2602.03489/) (2026)

## Abstract (arXiv)

Machine learning-based malware detectors are increasingly vulnerable to adversarial examples. Traditional defenses, such as one-shot adversarial training, often fail against adaptive attackers who use reinforcement learning to bypass detection. This paper proposes a robust defense framework based on bilevel optimization, explicitly modeling the strategic interaction between a defender and an attacker as an adversarial co-evolutionary process. We evaluate our approach using the MAB-malware framework against three distinct malware families: Mokes, Strab, and DCRat. Our experimental results demonstrate that while standard classifiers and basic adversarial retraining often remain vulnerable, showing evasion rates as high as 90 %, the proposed bilevel optimization approach consistently achieves near-total immunity, reducing evasion rates to 0 - 1.89 %. Furthermore, the iterative framework significantly increases the attacker's query complexity, raising the average cost of successful evasion by up to two orders of magnitude. These findings suggest that modeling the iterative cycle of attack and defense through bilevel optimization is essential for developing resilient malware detection systems capable of withstanding evolving adversarial threats.