# Automated Security Response through Online Learning with Adaptive Conjectures

arXiv: https://arxiv.org/abs/2402.12499 · HTML: https://mlsanity.com/paper/2402.12499/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 114 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Deep Reinforcement Learning for Cyber Security](https://mlsanity.com/paper/1906.05799/) (2019)
- [Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement](https://mlsanity.com/paper/1604.08127/) (2016)
- [Finding Effective Security Strategies through Reinforcement Learning and Self-Play](https://mlsanity.com/paper/2009.08120/) (2020)
- [Intrusion Prevention through Optimal Stopping](https://mlsanity.com/paper/2111.00289/) (2021)
- [A Dynamic Games Approach to Proactive Defense Strategies against Advanced Persistent Threats in Cyber-Physical Systems](https://mlsanity.com/paper/1906.09687/) (2019)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Solving Zero-Sum One-Sided Partially Observable Stochastic Games](https://mlsanity.com/paper/2010.11243/) (2020)
- [Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games](https://mlsanity.com/paper/2103.04539/) (2021)
- [Scenario-Agnostic Zero-Trust Defense with Explainable Threshold Policy: A Meta-Learning Approach](https://mlsanity.com/paper/2303.03349/) (2023)
- [Introducing a New Alert Data Set for Multi-Step Attack Analysis](https://mlsanity.com/paper/2308.12627/) (2023)
- [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2306.06766/) (2023)
- [Berk-Nash Equilibrium: A Framework for Modeling Agents with Misspecified Models](https://mlsanity.com/paper/1411.1152/) (2014)
- [Reinforcement Learning for POMDP: Partitioned Rollout and Policy Iteration with Application to Autonomous Sequential Repair Problems](https://mlsanity.com/paper/2002.04175/) (2020)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [A Game of Drones: Cyber-Physical Security of Time-Critical UAV Applications with Cumulative Prospect Theory Perceptions and Valuations](https://mlsanity.com/paper/1902.03506/) (2019)
- [Strategic Defense against Stealthy Link Flooding Attacks: A Signaling Game Approach](https://mlsanity.com/paper/1912.10073/) (2019)
- [Interdependent Strategic Security Risk Management with Bounded Rationality in the Internet of Things](https://mlsanity.com/paper/1905.09341/) (2019)
- [Behavioral and Game-Theoretic Security Investments in Interdependent Systems Modeled by Attack Graphs](https://mlsanity.com/paper/2001.03213/) (2020)
- [Burstiness of Intrusion Detection Process: Empirical Evidence and a Modeling Approach](https://mlsanity.com/paper/1707.03927/) (2017)
- [Causative Cyberattacks on Online Learning-based Automated Demand Response Systems](https://mlsanity.com/paper/2307.15175/) (2023)

## Built on by (most cited first, 13 of 13)

- [Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search](https://mlsanity.com/paper/2407.11070/) (2024)
- [Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination](https://mlsanity.com/paper/2508.05188/) (2025)
- [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025)
- [Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization](https://mlsanity.com/paper/2508.14385/) (2025)
- [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations](https://mlsanity.com/paper/2512.22883/) (2025)
- [Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model](https://mlsanity.com/paper/2509.06539/) (2025)
- [Hallucination-Resistant Security Planning with a Large Language Model](https://mlsanity.com/paper/2602.05279/) (2026)
- [Agentic Incident Response through Digital Twin-Enhanced Multiscale Planning](https://mlsanity.com/paper/2608.02422/) (2026)
- [STAIR: Effective Incident Response Using an End-to-End Agentic Planning Framework](https://mlsanity.com/paper/2608.09524/) (2026)
- [Optimal Stopping of Self-Refining Foundation Models](https://mlsanity.com/paper/2608.10729/) (2026)
- [Hierarchical Agentic Incident Response with Digital-Twin-Validated Attack Inference](https://mlsanity.com/paper/2608.15016/) (2026)

## Abstract (arXiv)

We study automated security response for an IT infrastructure and formulate the interaction between an attacker and a defender as a partially observed, non-stationary game. We relax the standard assumption that the game model is correctly specified and consider that each player has a probabilistic conjecture about the model, which may be misspecified in the sense that the true model has probability 0. This formulation allows us to capture uncertainty and misconception about the infrastructure and the intents of the players. To learn effective game strategies online, we design Conjectural Online Learning (COL), a novel method where a player iteratively adapts its conjecture using Bayesian learning and updates its strategy through rollout. We prove that the conjectures converge to best fits, and we provide a bound on the performance improvement that rollout enables with a conjectured model. To characterize the steady state of the game, we propose a variant of the Berk-Nash equilibrium. We present COL through an advanced persistent threat use case. Testbed evaluations show that COL produces effective security strategies that adapt to a changing environment. We also find that COL enables faster convergence than current reinforcement learning techniques.