# A Regularized Opponent Model with Maximum Entropy Objective

arXiv: https://arxiv.org/abs/1905.08087 · HTML: https://mlsanity.com/paper/1905.08087/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 10 of 10)

- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [Multi-Agent Imitation Learning with Copulas](https://mlsanity.com/paper/2107.04750/) (2021)
- [Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.10351/) (2023)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [Multi-agent Actor-Critic with Time Dynamical Opponent Model](https://mlsanity.com/paper/2204.05576/) (2022)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [Negotiated Reasoning: On Provably Addressing Relative Over-Generalization](https://mlsanity.com/paper/2306.05353/) (2023)
- [Active Legibility in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2410.20954/) (2024)
- [Learning to Infer Belief Embedded Communication](https://mlsanity.com/paper/2203.07832/) (2022)
- [Vairiational Stochastic Games](https://mlsanity.com/paper/2503.06037/) (2025)

## Abstract (arXiv)

In a single-agent setting, reinforcement learning (RL) tasks can be cast into an inference problem by introducing a binary random variable o, which stands for the "optimality". In this paper, we redefine the binary random variable o in multi-agent setting and formalize multi-agent reinforcement learning (MARL) as probabilistic inference. We derive a variational lower bound of the likelihood of achieving the optimality and name it as Regularized Opponent Model with Maximum Entropy Objective (ROMMEO). From ROMMEO, we present a novel perspective on opponent modeling and show how it can improve the performance of training agents theoretically and empirically in cooperative games. To optimize ROMMEO, we first introduce a tabular Q-iteration method ROMMEO-Q with proof of convergence. We extend the exact algorithm to complex environments by proposing an approximate version, ROMMEO-AC. We evaluate these two algorithms on the challenging iterated matrix game and differential game respectively and show that they can outperform strong MARL baselines.