# Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.11942 · HTML: https://mlsanity.com/paper/2210.11942/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Model-Free Opponent Shaping](https://mlsanity.com/paper/2205.01447/) (2022)
- [Sample-Efficient Learning of Stackelberg Equilibria in General-Sum Games](https://mlsanity.com/paper/2102.11494/) (2021)
- [Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?](https://mlsanity.com/paper/2112.13521/) (2021)
- [M3RL: Mind-aware Multi-agent Management Reinforcement Learning](https://mlsanity.com/paper/1810.00147/) (2018)
- [Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/2112.10859/) (2021)
- [Coordinating Followers to Reach Better Equilibria: End-to-End Gradient Descent for Stackelberg Games](https://mlsanity.com/paper/2106.03278/) (2021)
- [Learning to Mitigate AI Collusion on Economic Platforms](https://mlsanity.com/paper/2202.07106/) (2022)
- [The Good Shepherd: An Oracle Agent for Mechanism Design](https://mlsanity.com/paper/2202.10135/) (2022)
- [Reinforcement Learning of Sequential Price Mechanisms](https://mlsanity.com/paper/2010.01180/) (2020)
- [Finding Optimal Strategies in a Multi-Period Multi-Leader-Follower Stackelberg Game Using an Evolutionary Algorithm](https://mlsanity.com/paper/1307.6246/) (2013)
- [Solving Structured Hierarchical Games Using Differential Backward Induction](https://mlsanity.com/paper/2106.04663/) (2021)

## Built on by (most cited first, 12 of 12)

- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [Online Learning in Stackelberg Games with an Omniscient Follower](https://mlsanity.com/paper/2301.11518/) (2023)
- [Multi-Sender Persuasion: A Computational Perspective](https://mlsanity.com/paper/2402.04971/) (2024)
- [Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts](https://mlsanity.com/paper/2407.18074/) (2024)
- [Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL](https://mlsanity.com/paper/2305.17342/) (2023)
- [Neural Operators Can Play Dynamic Stackelberg Games](https://mlsanity.com/paper/2411.09644/) (2024)
- [CANDID DAC: Leveraging Coupled Action Dimensions with Importance Differences in DAC](https://mlsanity.com/paper/2407.05789/) (2024)
- [Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization](https://mlsanity.com/paper/2603.15388/) (2026)
- [Social Environment Design](https://mlsanity.com/paper/2402.14090/) (2024)
- [ReLExS: Reinforcement Learning Explanations for Stackelberg No-Regret Learners](https://mlsanity.com/paper/2408.14086/) (2024)
- [Large Legislative Models: Towards Efficient AI Policymaking in Economic Simulations](https://mlsanity.com/paper/2410.08345/) (2024)
- [Imitation Learning of Correlated Policies in Stackelberg Games](https://mlsanity.com/paper/2503.08883/) (2025)

## Abstract (arXiv)

Stackelberg equilibria arise naturally in a range of popular learning problems, such as in security games or indirect mechanism design, and have received increasing attention in the reinforcement learning literature. We present a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem, allowing a wide range of algorithmic design choices. We discuss how previous approaches can be seen as specific instantiations of this framework. As a key insight, we note that the design space allows for approaches not previously seen in the literature, for instance by leveraging multitask and meta-RL techniques for follower convergence. We propose one such approach using contextual policies, and evaluate it experimentally on both standard and novel benchmark domains, showing greatly improved sample efficiency compared to previous approaches. Finally, we explore the effect of adopting algorithm designs outside the borders of our framework.