# Best Response Shaping

arXiv: https://arxiv.org/abs/2404.06519 · HTML: https://mlsanity.com/paper/2404.06519/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 7 of 7)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Bi-level Actor-Critic for Multi-agent Coordination](https://mlsanity.com/paper/1909.03510/) (2019)
- [Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences](https://mlsanity.com/paper/2011.05373/) (2020)
- [The Good Shepherd: An Oracle Agent for Mechanism Design](https://mlsanity.com/paper/2202.10135/) (2022)

## Built on by (most cited first, 3 of 3)

- [LOQA: Learning with Opponent Q-Learning Awareness](https://mlsanity.com/paper/2405.01035/) (2024)
- [Advantage Alignment Algorithms](https://mlsanity.com/paper/2406.14662/) (2024)
- [Multi-agent cooperation through in-context co-player inference](https://mlsanity.com/paper/2602.16301/) (2026)

## Abstract (arXiv)

We investigate the challenge of multi-agent deep reinforcement learning in partially competitive environments, where traditional methods struggle to foster reciprocity-based cooperation. LOLA and POLA agents learn reciprocity-based cooperative policies by differentiation through a few look-ahead optimization steps of their opponent. However, there is a key limitation in these techniques. Because they consider a few optimization steps, a learning opponent that takes many steps to optimize its return may exploit them. In response, we introduce a novel approach, Best Response Shaping (BRS), which differentiates through an opponent approximating the best response, termed the "detective." To condition the detective on the agent's policy for complex games we propose a state-aware differentiable conditioning mechanism, facilitated by a question answering (QA) method that extracts a representation of the agent based on its behaviour on specific environment states. To empirically validate our method, we showcase its enhanced performance against a Monte Carlo Tree Search (MCTS) opponent, which serves as an approximation to the best response in the Coin Game. This work expands the applicability of multi-agent RL in partially competitive environments and provides a new pathway towards achieving improved social welfare in general sum games.