# Learning to Play against Any Mixture of Opponents

arXiv: https://arxiv.org/abs/2009.14180 · HTML: https://mlsanity.com/paper/2009.14180/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [Deep Interactive Bayesian Reinforcement Learning via Meta-Learning](https://mlsanity.com/paper/2101.03864/) (2021)
- [A Game-Theoretic Approach for Improving Generalization Ability of TSP Solvers](https://mlsanity.com/paper/2110.15105/) (2021)
- [NeuPL: Neural Population Learning](https://mlsanity.com/paper/2202.07415/) (2022)
- [Simplex Neural Population Learning: Any-Mixture Bayes-Optimality in Symmetric Zero-sum Games](https://mlsanity.com/paper/2205.15879/) (2022)
- [Generalized Beliefs for Cooperative AI](https://mlsanity.com/paper/2206.12765/) (2022)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense](https://mlsanity.com/paper/2508.19488/) (2025)
- [DNQ: Deep Nash Q-Network for Partially Observable n-Player Games](https://mlsanity.com/paper/2606.06480/) (2026)

## Abstract (arXiv)

Intuitively, experience playing against one mixture of opponents in a given domain should be relevant for a different mixture in the same domain. We propose a transfer learning method, Q-Mixing, that starts by learning Q-values against each pure-strategy opponent. Then a Q-value for any distribution of opponent strategies is approximated by appropriately averaging the separately learned Q-values. From these components, we construct policies against all opponent mixtures without any further training. We empirically validate Q-Mixing in two environments: a simple grid-world soccer environment, and a complicated cyber-security game. We find that Q-Mixing is able to successfully transfer knowledge across any mixture of opponents. We next consider the use of observations during play to update the believed distribution of opponents. We introduce an opponent classifier -- trained in parallel to Q-learning, using the same data -- and use the classifier results to refine the mixing of Q-values. We find that Q-Mixing augmented with the opponent classifier function performs comparably, and with lower variance, than training directly against a mixed-strategy opponent.