# Multiplayer Bandit Learning, from Competition to Cooperation

arXiv: https://arxiv.org/abs/1908.01135 · HTML: https://mlsanity.com/paper/1908.01135/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2019.

## Built on by (most cited first, 12 of 12)

- [Federated Multi-armed Bandits with Personalization](https://mlsanity.com/paper/2102.13101/) (2021)
- [Beyond log2(T) Regret for Decentralized Bandits in Matching Markets](https://mlsanity.com/paper/2103.07501/) (2021)
- [A survey on multi-player bandits](https://mlsanity.com/paper/2211.16275/) (2022)
- [Competition, Alignment, and Equilibria in Digital Marketplaces](https://mlsanity.com/paper/2208.14423/) (2022)
- [Trustworthy Distributed AI Systems: Robustness, Privacy, and Governance](https://mlsanity.com/paper/2402.01096/) (2024)
- [Competing for Shareable Arms in Multi-Player Multi-Armed Bandits](https://mlsanity.com/paper/2305.19158/) (2023)
- [Multiplayer Information Asymmetric Contextual Bandits](https://mlsanity.com/paper/2503.08961/) (2025)
- [Optimistic Thompson Sampling for No-Regret Learning in Unknown Games](https://mlsanity.com/paper/2402.09456/) (2024)
- [PPA-Game: Characterizing and Learning Competitive Dynamics Among Online Content Creators](https://mlsanity.com/paper/2403.15524/) (2024)
- [Competitive Multi-armed Bandit Games for Resource Sharing](https://mlsanity.com/paper/2503.20975/) (2025)
- [Bayesian Decision Making around Experts](https://mlsanity.com/paper/2510.08113/) (2025)
- [DCM Bandits: Multiplayer Information Asymmetric Cascading Bandits for Multiple Clicks](https://mlsanity.com/paper/2608.11873/) (2026)

## Abstract (arXiv)

The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are k arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is GammaA + λ GammaB (and similarly for Bob), where GammaA is Alice's total reward and λ ∈ [-1, 1] is a cooperation parameter. At λ = -1 the players are competing in a zero-sum game, at λ = 1, they are fully cooperating, and at λ = 0, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards. With discount factor β, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior μ, and a predictable arm, with success probability p. The value of p where the player is indifferent between the arms is the Gittins index g = g(μ,β) > m, where m is the mean of the risky arm. We show that competing players explore less than a single player: there is p^* ∈ (m, g) so that for all p > p^*, the players stay at the predictable arm. However, the players are not myopic: they still explore for some p > m. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all  p∈ (p^*, g), where p^* is the threshold from the competing case. Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.