# Temporal Induced Self-Play for Stochastic Bayesian Games

arXiv: https://arxiv.org/abs/2108.09444 · HTML: https://mlsanity.com/paper/2108.09444/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 21 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1811.01458/) (2018)
- [A Game-Theoretic Model and Best-Response Learning Method for Ad Hoc Coordination in Multiagent Systems](https://mlsanity.com/paper/1506.01170/) (2015)
- [Finding Friend and Foe in Multi-Agent Games](https://mlsanity.com/paper/1906.02330/) (2019)
- [Human-Level Performance in No-Press Diplomacy via Equilibrium Search](https://mlsanity.com/paper/2010.02923/) (2020)
- [Belief and Truth in Hypothesised Behaviours](https://mlsanity.com/paper/1507.07688/) (2015)
- [Learning to Interactively Learn and Assist](https://mlsanity.com/paper/1906.10187/) (2019)
- [Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games](https://mlsanity.com/paper/1909.08735/) (2019)
- [Smoothing Method for Approximate Extensive-Form Perfect Equilibrium](https://mlsanity.com/paper/1705.09326/) (2017)
- [Regret Minimization in Behaviorally-Constrained Zero-Sum Games](https://mlsanity.com/paper/1711.03441/) (2017)
- [Extensive-Form Perfect Equilibrium Computation in Two-Player Games](https://mlsanity.com/paper/1611.05011/) (2016)
- [The complexity of approximating a trembling hand perfect equilibrium of a multi-player game in strategic form](https://mlsanity.com/paper/1408.1017/) (2014)

## Built on by (most cited first, 2 of 2)

- [Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning](https://mlsanity.com/paper/2310.04796/) (2023)
- [MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search](https://mlsanity.com/paper/2404.03101/) (2024)

## Abstract (arXiv)

One practical requirement in solving dynamic games is to ensure that the players play well from any decision point onward. To satisfy this requirement, existing efforts focus on equilibrium refinement, but the scalability and applicability of existing techniques are limited. In this paper, we propose Temporal-Induced Self-Play (TISP), a novel reinforcement learning-based framework to find strategies with decent performances from any decision point onward. TISP uses belief-space representation, backward induction, policy learning, and non-parametric approximation. Building upon TISP, we design a policy-gradient-based algorithm TISP-PG. We prove that TISP-based algorithms can find approximate Perfect Bayesian Equilibrium in zero-sum one-sided stochastic Bayesian games with finite horizon. We test TISP-based algorithms in various games, including finitely repeated security games and a grid-world game. The results show that TISP-PG is more scalable than existing mathematical programming-based methods and significantly outperforms other learning-based methods.