# Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games

arXiv: https://arxiv.org/abs/2402.18781 · HTML: https://mlsanity.com/paper/2402.18781/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement](https://mlsanity.com/paper/1604.08127/) (2016)
- [The Confluence of Networks, Games and Learning](https://mlsanity.com/paper/2105.08158/) (2021)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Solving Zero-Sum One-Sided Partially Observable Stochastic Games](https://mlsanity.com/paper/2010.11243/) (2020)
- [On the Price of Transparency: A Comparison between Overt Persuasion and Covert Signaling](https://mlsanity.com/paper/2304.00096/) (2023)
- [Berk-Nash Equilibrium: A Framework for Modeling Agents with Misspecified Models](https://mlsanity.com/paper/1411.1152/) (2014)
- [HSVI can solve zero-sum Partially Observable Stochastic Games](https://mlsanity.com/paper/2210.14640/) (2022)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [Computing Approximate Nash Equilibria and Robust Best-Responses Using Sampling](https://mlsanity.com/paper/1401.4591/) (2014)
- [Decision-Dominant Strategic Defense Against Lateral Movement for 5G Zero-Trust Multi-Domain Networks](https://mlsanity.com/paper/2310.01675/) (2023)
- [Dynamic Games with Asymmetric Information: Common Information Based Perfect Bayesian Equilibria and Sequential Decomposition](https://mlsanity.com/paper/1510.07001/) (2015)
- [Hidden Stochastic Games and Limit Equilibrium Payoffs](https://mlsanity.com/paper/1407.3028/) (2014)
- [Subjective Equilibria under Beliefs of Exogenous Uncertainty for Dynamic Games](https://mlsanity.com/paper/2005.01640/) (2020)

## Built on by (most cited first, 14 of 14)

- [Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search](https://mlsanity.com/paper/2407.11070/) (2024)
- [Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination](https://mlsanity.com/paper/2508.05188/) (2025)
- [Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks](https://mlsanity.com/paper/2410.17431/) (2024)
- [Symbiotic Game and Foundation Models for Cyber Deception Operations in Strategic Cyber Warfare](https://mlsanity.com/paper/2403.10570/) (2024)
- [Intrusion Tolerance for Networked Systems through Two-Level Feedback Control](https://mlsanity.com/paper/2404.01741/) (2024)
- [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations](https://mlsanity.com/paper/2512.22883/) (2025)
- [In-Context Autonomous Network Incident Response: An End-to-End Large Language Model Agent Approach](https://mlsanity.com/paper/2602.13156/) (2026)
- [Agentic Incident Response through Digital Twin-Enhanced Multiscale Planning](https://mlsanity.com/paper/2608.02422/) (2026)
- [The Game-Theoretic Symbiosis of Trust and AI in Networked Systems](https://mlsanity.com/paper/2411.12859/) (2024)
- [Hierarchical Agentic Incident Response with Digital-Twin-Validated Attack Inference](https://mlsanity.com/paper/2608.15016/) (2026)

## Abstract (arXiv)

Asymmetric information stochastic games (AISGs) arise in many complex socio-technical systems, such as cyber-physical systems and IT infrastructures. Existing computational methods for AISGs are primarily offline and can not adapt to equilibrium deviations. Further, current methods are limited to particular information structures to avoid belief hierarchies. Considering these limitations, we propose conjectural online learning (COL), an online learning method under generic information structures in AISGs. COL uses a forecaster-actor-critic (FAC) architecture, where subjective forecasts are used to conjecture the opponents' strategies within a lookahead horizon, and Bayesian learning is used to calibrate the conjectures. To adapt strategies to nonstationary environments based on information feedback, COL uses online rollout with cost function approximation (actor-critic). We prove that the conjectures produced by COL are asymptotically consistent with the information feedback in the sense of a relaxed Bayesian consistency. We also prove that the empirical strategy profile induced by COL converges to the Berk-Nash equilibrium, a solution concept characterizing rationality under subjectivity. Experimental results from an intrusion response use case demonstrate COL's {faster convergence} over state-of-the-art reinforcement learning methods against nonstationary attacks.