# Learning Near-Optimal Intrusion Responses Against Dynamic Attackers

arXiv: https://arxiv.org/abs/2301.06085 · HTML: https://mlsanity.com/paper/2301.06085/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 191 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 30 of 35)

- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [CybORG: A Gym for the Development of Autonomous Cyber Agents](https://mlsanity.com/paper/2108.09118/) (2021)
- [Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation](https://mlsanity.com/paper/2207.12355/) (2022)
- [Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement](https://mlsanity.com/paper/1604.08127/) (2016)
- [Finding Effective Security Strategies through Reinforcement Learning and Self-Play](https://mlsanity.com/paper/2009.08120/) (2020)
- [Network Environment Design for Autonomous Cyberdefense](https://mlsanity.com/paper/2103.07583/) (2021)
- [Reinforcement Learning for IoT Security: A Comprehensive Survey](https://mlsanity.com/paper/2102.07247/) (2021)
- [Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies](https://mlsanity.com/paper/2211.15557/) (2022)
- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Intrusion Prevention through Optimal Stopping](https://mlsanity.com/paper/2111.00289/) (2021)
- [Reinforcement Learning for Autonomous Defence in Software-Defined Networking](https://mlsanity.com/paper/1808.05770/) (2018)
- [A Dynamic Games Approach to Proactive Defense Strategies against Advanced Persistent Threats in Cyber-Physical Systems](https://mlsanity.com/paper/1906.09687/) (2019)
- [Using Cyber Terrain in Reinforcement Learning for Penetration Testing](https://mlsanity.com/paper/2108.07124/) (2021)
- [Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach](https://mlsanity.com/paper/1809.05258/) (2018)
- [Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense](https://mlsanity.com/paper/2007.10457/) (2020)
- [Solving Zero-Sum One-Sided Partially Observable Stochastic Games](https://mlsanity.com/paper/2010.11243/) (2020)
- [Prospective Artificial Intelligence Approaches for Active Cyber Defence](https://mlsanity.com/paper/2104.09981/) (2021)
- [Learning Intrusion Prevention Policies through Optimal Stopping](https://mlsanity.com/paper/2106.07160/) (2021)
- [CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems](https://mlsanity.com/paper/2109.03331/) (2021)
- [How to Attack and Defend NextG Radio Access Network Slicing with Reinforcement Learning](https://mlsanity.com/paper/2101.05768/) (2021)
- [Solving Large-Scale Extensive-Form Network Security Games via Neural Fictitious Self-Play](https://mlsanity.com/paper/2106.00897/) (2021)
- [A Supervisory Control Approach to Dynamic Cyber-Security](https://mlsanity.com/paper/1409.0838/) (2014)
- [Learning Security Strategies through Game Play and Optimal Stopping](https://mlsanity.com/paper/2205.14694/) (2022)
- [Multiple Stopping Time POMDPs: Structural Results & Application in Interactive Advertising in Social Media](https://mlsanity.com/paper/1712.00524/) (2017)
- [Strategic Defense against Stealthy Link Flooding Attacks: A Signaling Game Approach](https://mlsanity.com/paper/1912.10073/) (2019)
- [Directional Laplacian Centrality for Cyber Situational Awareness](https://mlsanity.com/paper/2008.04357/) (2020)
- [Constraints Satisfiability Driven Reinforcement Learning for Autonomous Cyber Defense](https://mlsanity.com/paper/2104.08994/) (2021)

## Built on by (most cited first, 15 of 15)

- [Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games](https://mlsanity.com/paper/2402.18781/) (2024)
- [Automated Security Response through Online Learning with Adaptive Conjectures](https://mlsanity.com/paper/2402.12499/) (2024)
- [Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search](https://mlsanity.com/paper/2407.11070/) (2024)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [Intrusion Tolerance for Networked Systems through Two-Level Feedback Control](https://mlsanity.com/paper/2404.01741/) (2024)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model](https://mlsanity.com/paper/2509.06539/) (2025)
- [IT Intrusion Detection Using Statistical Learning and Testbed Measurements](https://mlsanity.com/paper/2402.13081/) (2024)
- [Training RL Agents for Multi-Objective Network Defense Tasks](https://mlsanity.com/paper/2505.22531/) (2025)
- [STAIR: Effective Incident Response Using an End-to-End Agentic Planning Framework](https://mlsanity.com/paper/2608.09524/) (2026)
- [Adaptive Learning for Moving Target defence: Enhancing Cybersecurity Strategies](https://mlsanity.com/paper/2508.17945/) (2025)
- [Online Identification of IT Systems through Active Causal Learning](https://mlsanity.com/paper/2509.02130/) (2025)
- [A Modular Framework for Rapidly Building Intrusion Predictors](https://mlsanity.com/paper/2511.23000/) (2025)
- [The MAL Simulator: Cyber Operations Simulation based on Attack & Defense Graphs](https://mlsanity.com/paper/2609.16563/) (2026)
- [Agentic-ZTA: A Multi-Agent Architecture for Autonomous Zero Trust Enforcement](https://mlsanity.com/paper/2610.05782/) (2026)

## Abstract (arXiv)

We study automated intrusion response and formulate the interaction between an attacker and a defender as an optimal stopping game where attack and defense strategies evolve through reinforcement learning and self-play. The game-theoretic modeling enables us to find defender strategies that are effective against a dynamic attacker, i.e. an attacker that adapts its strategy in response to the defender strategy. Further, the optimal stopping formulation allows us to prove that optimal strategies have threshold properties. To obtain near-optimal defender strategies, we develop Threshold Fictitious Self-Play (T-FP), a fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that T-FP outperforms a state-of-the-art algorithm for our use case. The experimental part of this investigation includes two systems: a simulation system where defender strategies are incrementally learned and an emulation system where statistics are collected that drive simulation runs and where learned strategies are evaluated. We argue that this approach can produce effective defender strategies for a practical IT infrastructure.