# Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense

arXiv: https://arxiv.org/abs/2007.10457 · HTML: https://mlsanity.com/paper/2007.10457/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Learning to Mitigate AI Collusion on Economic Platforms](https://mlsanity.com/paper/2202.07106/) (2022)
- [Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks](https://mlsanity.com/paper/2410.17431/) (2024)
- [The Adaptive Arms Race: Redefining Robustness in AI Security](https://mlsanity.com/paper/2312.13435/) (2023)
- [RL and Fingerprinting to Select Moving Target Defense Mechanisms for Zero-day Attacks in IoT](https://mlsanity.com/paper/2212.14647/) (2022)
- [The Role of Deep Learning in Advancing Proactive Cybersecurity Measures for Smart Grid Networks: A Survey](https://mlsanity.com/paper/2401.05896/) (2024)
- [CyberForce: A Federated Reinforcement Learning Framework for Malware Mitigation](https://mlsanity.com/paper/2308.05978/) (2023)
- [Survey on Multi-Agent Q-Learning frameworks for resource management in wireless sensor network](https://mlsanity.com/paper/2105.02371/) (2021)
- [Pandering in a Flexible Representative Democracy](https://mlsanity.com/paper/2211.09986/) (2022)
- [Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty](https://mlsanity.com/paper/2305.16469/) (2023)
- [VISER: A Tractable Solution Concept for Games with Information Asymmetry](https://mlsanity.com/paper/2307.09652/) (2023)
- [A Factored MDP Approach To Moving Target Defense With Dynamic Threat Modeling and Cost Efficiency](https://mlsanity.com/paper/2408.08934/) (2024)
- [Adaptive Learning for Moving Target defence: Enhancing Cybersecurity Strategies](https://mlsanity.com/paper/2508.17945/) (2025)
- [Robust Strategic Classification under Decision-Dependent Cost Uncertainty](https://mlsanity.com/paper/2606.30136/) (2026)

## Abstract (arXiv)

The field of cybersecurity has mostly been a cat-and-mouse game with the discovery of new attacks leading the way. To take away an attacker's advantage of reconnaissance, researchers have proposed proactive defense methods such as Moving Target Defense (MTD). To find good movement strategies, researchers have modeled MTD as leader-follower games between the defender and a cyber-adversary. We argue that existing models are inadequate in sequential settings when there is incomplete information about a rational adversary and yield sub-optimal movement strategies. Further, while there exists an array of work on learning defense policies in sequential settings for cyber-security, they are either unpopular due to scalability issues arising out of incomplete information or tend to ignore the strategic nature of the adversary simplifying the scenario to use single-agent reinforcement learning techniques. To address these concerns, we propose (1) a unifying game-theoretic model, called the Bayesian Stackelberg Markov Games (BSMGs), that can model uncertainty over attacker types and the nuances of an MTD system and (2) a Bayesian Strong Stackelberg Q-learning (BSS-Q) approach that can, via interaction, learn the optimal movement policy for BSMGs within a reasonable time. We situate BSMGs in the landscape of incomplete-information Markov games and characterize the notion of Strong Stackelberg Equilibrium (SSE) in them. We show that our learning approach converges to an SSE of a BSMG and then highlight that the learned movement policy (1) improves the state-of-the-art in MTD for web-application security and (2) converges to an optimal policy in MTD domains with incomplete information about adversaries even when prior information about rewards and transitions is absent.