# Learning Security Strategies through Game Play and Optimal Stopping

arXiv: https://arxiv.org/abs/2205.14694 · HTML: https://mlsanity.com/paper/2205.14694/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 109 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [CybORG: A Gym for the Development of Autonomous Cyber Agents](https://mlsanity.com/paper/2108.09118/) (2021)
- [Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement](https://mlsanity.com/paper/1604.08127/) (2016)
- [Finding Effective Security Strategies through Reinforcement Learning and Self-Play](https://mlsanity.com/paper/2009.08120/) (2020)
- [Network Environment Design for Autonomous Cyberdefense](https://mlsanity.com/paper/2103.07583/) (2021)
- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Intrusion Prevention through Optimal Stopping](https://mlsanity.com/paper/2111.00289/) (2021)
- [A Dynamic Games Approach to Proactive Defense Strategies against Advanced Persistent Threats in Cyber-Physical Systems](https://mlsanity.com/paper/1906.09687/) (2019)
- [Using Cyber Terrain in Reinforcement Learning for Penetration Testing](https://mlsanity.com/paper/2108.07124/) (2021)
- [Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach](https://mlsanity.com/paper/1809.05258/) (2018)
- [Solving Zero-Sum One-Sided Partially Observable Stochastic Games](https://mlsanity.com/paper/2010.11243/) (2020)
- [Prospective Artificial Intelligence Approaches for Active Cyber Defence](https://mlsanity.com/paper/2104.09981/) (2021)
- [Learning Intrusion Prevention Policies through Optimal Stopping](https://mlsanity.com/paper/2106.07160/) (2021)
- [CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems](https://mlsanity.com/paper/2109.03331/) (2021)
- [A Supervisory Control Approach to Dynamic Cyber-Security](https://mlsanity.com/paper/1409.0838/) (2014)
- [Multiple Stopping Time POMDPs: Structural Results & Application in Interactive Advertising in Social Media](https://mlsanity.com/paper/1712.00524/) (2017)
- [Strategic Defense against Stealthy Link Flooding Attacks: A Signaling Game Approach](https://mlsanity.com/paper/1912.10073/) (2019)
- [Constraints Satisfiability Driven Reinforcement Learning for Autonomous Cyber Defense](https://mlsanity.com/paper/2104.08994/) (2021)
- [A System for Interactive Examination of Learned Security Policies](https://mlsanity.com/paper/2204.01126/) (2022)
- [Multiple Domain Cyberspace Attack and Defense Game Based on Reward Randomization Reinforcement Learning](https://mlsanity.com/paper/2205.10990/) (2022)

## Built on by (most cited first, 5 of 5)

- [Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies](https://mlsanity.com/paper/2211.15557/) (2022)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Entity-based Reinforcement Learning for Autonomous Cyber Defence](https://mlsanity.com/paper/2410.17647/) (2024)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [IT Intrusion Detection Using Statistical Learning and Testbed Measurements](https://mlsanity.com/paper/2402.13081/) (2024)

## Abstract (arXiv)

We study automated intrusion prevention using reinforcement learning. Following a novel approach, we formulate the interaction between an attacker and a defender as an optimal stopping game and let attack and defense strategies evolve through reinforcement learning and self-play. The game-theoretic perspective allows us to find defender strategies that are effective against dynamic attackers. The optimal stopping formulation gives us insight into the structure of optimal strategies, which we show to have threshold properties. To obtain the optimal defender strategies, we introduce T-FP, a fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that T-FP outperforms a state-of-the-art algorithm for our use case. Our overall method for learning and evaluating strategies includes two systems: a simulation system where defender strategies are incrementally learned and an emulation system where statistics are produced that drive simulation runs and where learned strategies are evaluated. We conclude that this approach can produce effective defender strategies for a practical IT infrastructure.