# Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

arXiv: https://arxiv.org/abs/1909.08735 · HTML: https://mlsanity.com/paper/1909.08735/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [Robust Multi-Agent Reinforcement Learning with State Uncertainty](https://mlsanity.com/paper/2307.16212/) (2023)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Evaluating the Robustness of Collaborative Agents](https://mlsanity.com/paper/2101.05507/) (2021)
- [Ensemble Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.02618/) (2023)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation](https://mlsanity.com/paper/2405.11467/) (2024)
- [Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective](https://mlsanity.com/paper/2106.05402/) (2021)
- [Temporal Induced Self-Play for Stochastic Bayesian Games](https://mlsanity.com/paper/2108.09444/) (2021)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [AI in Human-computer Gaming: Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2111.07631/) (2021)
- [Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games](https://mlsanity.com/paper/2411.04976/) (2024)

## Abstract (arXiv)

This paper presents an algorithmic framework for learning robust policies in asymmetric imperfect-information games, where the joint reward could depend on the uncertain opponent type (a private information known only to the opponent itself and its ally). In order to maximize the reward, the protagonist agent has to infer the opponent type through agent modeling. We use multiagent reinforcement learning (MARL) to learn opponent models through self-play, which captures the full strategy interaction and reasoning between agents. However, agent policies learned from self-play can suffer from mutual overfitting. Ensemble training methods can be used to improve the robustness of agent policy against different opponents, but it also significantly increases the computational overhead. In order to achieve a good trade-off between the robustness of the learned policy and the computation complexity, we propose to train a separate opponent policy against the protagonist agent for evaluation purposes. The reward achieved by this opponent is a noisy measure of the robustness of the protagonist agent policy due to the intrinsic stochastic nature of a reinforcement learner. To handle this stochasticity, we apply a stochastic optimization scheme to dynamically update the opponent ensemble to optimize an objective function that strikes a balance between robustness and computation complexity. We empirically show that, under the same limited computational budget, the proposed method results in more robust policy learning than standard ensemble training.