mlsanityData to 7 Oct 2026

SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity

arXiv 2602.08690 ↗First posted 9 Feb 2026cs.LG cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (DRL) has achieved remarkable success in domains requiring sequential decision-making, motivating its application to cybersecurity problems. However, transitioning DRL from laboratory simulations to bespoke cyber environments can introduce numerous issues. This is further exacerbated by the often adversarial, non-stationary, and partially-observable nature of most cybersecurity tasks. In this paper, we identify and systematize 11 methodological pitfalls that frequently occur in DRL for cybersecurity (DRL4Sec) literature across the stages of environment modeling, agent training, performance evaluation, and system deployment. By analyzing 66 significant DRL4Sec papers (2018-2025), we quantify the prevalence of each pitfall and find an average of over five pitfalls per paper. We demonstrate the practical impact of these pitfalls using controlled experiments in (i) autonomous cyber defense, (ii) adversarial malware creation, and (iii) web security testing environments. Finally, we provide actionable recommendations for each pitfall to support the development of more rigorous and deployable DRL-based security systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 75 of its 155 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

75 of 155 references.

References · 75

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  12. 2017Deep Reinforcement Learning that Matterscited by 727
38 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 518
  2. 2019Model-Based Reinforcement Learning for Ataricited by 495
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  4. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2019Challenges of Real-World Reinforcement Learningcited by 282
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  8. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  9. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  10. 2020Neural Thompson Samplingcited by 127
  11. 2018TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Timecited by 108
  12. 2020Dos and Don'ts of Machine Learning in Computer Securitycited by 98
  13. 2019Deep Reinforcement Learning for Cyber Securitycited by 92
  14. 2020PatchAttack: A Black-box Texture-based Attack with Reinforcement Learningcited by 67
  15. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  16. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62
  17. 2018Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learningcited by 60
  18. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  19. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56
  20. 2019Stateful Detection of Black-Box Adversarial Attackscited by 55
  21. 2023Empirical Design in Reinforcement Learningcited by 54
  22. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  23. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 49
  24. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 42
  25. 2022Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulationcited by 37
  26. 2018Evaluating Fuzz Testingcited by 29
  27. 2024Achieving Human Level Competitive Robot Table Tenniscited by 24
  28. 2019On Inductive Biases in Deep Reinforcement Learningcited by 22
  29. 2023Reward Shaping for Happier Autonomous Cyber Security Agentscited by 19
  30. 2018Reinforcement Learning for Autonomous Defence in Software-Defined Networkingcited by 17
  31. 2023Inroads into Autonomous Network Defence using Explained Reinforcement Learningcited by 17
  32. 2021Using Cyber Terrain in Reinforcement Learning for Penetration Testingcited by 16
  33. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learningcited by 15
  34. 2018Deep Reinforcement Fuzzingcited by 14
  35. 2024Autonomous Network Defence using Reinforcement Learningcited by 14
  36. 2019Finding Needles in a Moving Haystack: Prioritizing Alerts with Adversarial Reinforcement Learningcited by 12
  37. 2022ATTRITION: Attacking Static Hardware Trojan Detection Techniques Using Reinforcement Learningcited by 11
  38. 2024CybORG++: An Enhanced Gym for the Development of Autonomous Cyber Agentscited by 11

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Building Better Environments for Autonomous Cyber Defencecited by 3
  2. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  3. 2026REPLICANT: Learning Policies for Evading and Hardening Malware Detectorscited by 0
  4. 2026CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defensecited by 0