# SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity

arXiv: https://arxiv.org/abs/2602.08690 · HTML: https://mlsanity.com/paper/2602.08690/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 75 of its 155 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 30 of 75)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Neural Thompson Sampling](https://mlsanity.com/paper/2010.00827/) (2020)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning](https://mlsanity.com/paper/2004.05682/) (2020)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning](https://mlsanity.com/paper/2005.07099/) (2020)
- [Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning](https://mlsanity.com/paper/1801.08917/) (2018)
- [How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments](https://mlsanity.com/paper/1806.08295/) (2018)

## Built on by (most cited first, 4 of 4)

- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)](https://mlsanity.com/paper/2608.04317/) (2026)
- [REPLICANT: Learning Policies for Evading and Hardening Malware Detectors](https://mlsanity.com/paper/2608.28499/) (2026)
- [CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defense](https://mlsanity.com/paper/2609.31893/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) has achieved remarkable success in domains requiring sequential decision-making, motivating its application to cybersecurity problems. However, transitioning DRL from laboratory simulations to bespoke cyber environments can introduce numerous issues. This is further exacerbated by the often adversarial, non-stationary, and partially-observable nature of most cybersecurity tasks. In this paper, we identify and systematize 11 methodological pitfalls that frequently occur in DRL for cybersecurity (DRL4Sec) literature across the stages of environment modeling, agent training, performance evaluation, and system deployment. By analyzing 66 significant DRL4Sec papers (2018-2025), we quantify the prevalence of each pitfall and find an average of over five pitfalls per paper. We demonstrate the practical impact of these pitfalls using controlled experiments in (i) autonomous cyber defense, (ii) adversarial malware creation, and (iii) web security testing environments. Finally, we provide actionable recommendations for each pitfall to support the development of more rigorous and deployable DRL-based security systems.