# Intrusion Prevention through Optimal Stopping

arXiv: https://arxiv.org/abs/2111.00289 · HTML: https://mlsanity.com/paper/2111.00289/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 103 references that are in the map.
- First posted on arXiv 30 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [CybORG: A Gym for the Development of Autonomous Cyber Agents](https://mlsanity.com/paper/2108.09118/) (2021)
- [Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement](https://mlsanity.com/paper/1604.08127/) (2016)
- [Finding Effective Security Strategies through Reinforcement Learning and Self-Play](https://mlsanity.com/paper/2009.08120/) (2020)
- [Network Environment Design for Autonomous Cyberdefense](https://mlsanity.com/paper/2103.07583/) (2021)
- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Using Cyber Terrain in Reinforcement Learning for Penetration Testing](https://mlsanity.com/paper/2108.07124/) (2021)
- [Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach](https://mlsanity.com/paper/1809.05258/) (2018)
- [Prospective Artificial Intelligence Approaches for Active Cyber Defence](https://mlsanity.com/paper/2104.09981/) (2021)
- [Learning Intrusion Prevention Policies through Optimal Stopping](https://mlsanity.com/paper/2106.07160/) (2021)
- [Adaptive Honeypot Engagement through Reinforcement Learning of Semi-Markov Decision Processes](https://mlsanity.com/paper/1906.12182/) (2019)
- [Online Reinforcement Learning of Optimal Threshold Policies for Markov Decision Processes](https://mlsanity.com/paper/1912.10325/) (2019)
- [A Supervisory Control Approach to Dynamic Cyber-Security](https://mlsanity.com/paper/1409.0838/) (2014)
- [Multiple Stopping Time POMDPs: Structural Results & Application in Interactive Advertising in Social Media](https://mlsanity.com/paper/1712.00524/) (2017)

## Built on by (most cited first, 18 of 18)

- [Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation](https://mlsanity.com/paper/2207.12355/) (2022)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Automated Security Response through Online Learning with Adaptive Conjectures](https://mlsanity.com/paper/2402.12499/) (2024)
- [Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search](https://mlsanity.com/paper/2407.11070/) (2024)
- [Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination](https://mlsanity.com/paper/2508.05188/) (2025)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [Intrusion Tolerance for Networked Systems through Two-Level Feedback Control](https://mlsanity.com/paper/2404.01741/) (2024)
- [Learning Security Strategies through Game Play and Optimal Stopping](https://mlsanity.com/paper/2205.14694/) (2022)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [A System for Interactive Examination of Learned Security Policies](https://mlsanity.com/paper/2204.01126/) (2022)
- [Building Better Environments for Autonomous Cyber Defence](https://mlsanity.com/paper/2604.08805/) (2026)
- [Dynamically meeting performance objectives for multiple services on a service mesh](https://mlsanity.com/paper/2210.04002/) (2022)
- [A Survey for Deep Reinforcement Learning Based Network Intrusion Detection](https://mlsanity.com/paper/2410.07612/) (2024)
- [Optimal Observation-Intervention Trade-Off in Optimisation Problems with Causal Structure](https://mlsanity.com/paper/2309.02287/) (2023)
- [Online Identification of IT Systems through Active Causal Learning](https://mlsanity.com/paper/2509.02130/) (2025)
- [A Modular Framework for Rapidly Building Intrusion Predictors](https://mlsanity.com/paper/2511.23000/) (2025)
- [Optimal Stopping of Self-Refining Foundation Models](https://mlsanity.com/paper/2608.10729/) (2026)
- [The MAL Simulator: Cyber Operations Simulation based on Attack & Defense Graphs](https://mlsanity.com/paper/2609.16563/) (2026)

## Abstract (arXiv)

We study automated intrusion prevention using reinforcement learning. Following a novel approach, we formulate the problem of intrusion prevention as an (optimal) multiple stopping problem. This formulation gives us insight into the structure of optimal policies, which we show to have threshold properties. For most practical cases, it is not feasible to obtain an optimal defender policy using dynamic programming. We therefore develop a reinforcement learning approach to approximate an optimal threshold policy. We introduce T-SPSA, an efficient reinforcement learning algorithm that learns threshold policies through stochastic approximation. We show that T-SPSA outperforms state-of-the-art algorithms for our use case. Our overall method for learning and validating policies includes two systems: a simulation system where defender policies are incrementally learned and an emulation system where statistics are produced that drive simulation runs and where learned policies are evaluated. We show that this approach can produce effective defender policies for a practical IT infrastructure.