mlsanityData to 14 Oct 2026

Differentiable Arbitrating in Zero-sum Markov Games

arXiv 2302.10058 ↗First posted 20 Feb 2023cs.MA cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We initiate the study of how to perturb the reward in a zero-sum Markov game with two players to induce a desirable Nash equilibrium, namely arbitrating. Such a problem admits a bi-level optimization formulation. The lower level requires solving the Nash equilibrium under a given reward function, which makes the overall problem challenging to optimize in an end-to-end way. We propose a backpropagation scheme that differentiates through the Nash equilibrium, which provides the gradient feedback for the upper level. In particular, our method only requires a black-box solver for the (regularized) Nash equilibrium (NE). We develop the convergence analysis for the proposed framework with proper black-box NE solvers and demonstrate the empirical successes in two multi-agent reinforcement learning (MARL) environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 97 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

40 of 97 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  11. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  12. 2018Understanding the impact of entropy on policy optimizationcited by 176
28 more
  1. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  2. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  3. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  4. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  5. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 61
  6. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  7. 2020Learning to Incentivize Other Learning Agentscited by 57
  8. 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 51
  9. 2018Re-evaluating Evaluationcited by 47
  10. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
  11. 2016Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimizationcited by 37
  12. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 37
  13. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  14. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  15. 2021Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?cited by 24
  16. 2018On the Generalization of Stochastic Gradient Descent with Momentumcited by 20
  17. 2020End-to-End Learning and Intervention in Gamescited by 20
  18. 2020Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessiancited by 20
  19. 2021Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learningcited by 19
  20. 2021Provably Efficient Policy Optimization for Two-Player Zero-Sum Markov Gamescited by 18
  21. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 18
  22. 2018Learning to Defend by Learning to Attackcited by 17
  23. 2019Coordinating the Crowd: Inducing Desirable Equilibria in Non-Cooperative Systemscited by 12
  24. 2021Coordinating Followers to Reach Better Equilibria: End-to-End Gradient Descent for Stackelberg Gamescited by 10
  25. 2018Decentralized Accelerated Gradient Methods With Increasing Penalty Parameterscited by 8
  26. 2021An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learningcited by 7
  27. 2021Efficient, Fair, and Incentive-Compatible Healthcare Rationingcited by 4
  28. 2021The exact worst-case convergence rate of the gradient method with fixed step lengths for L-smooth functionscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Learning to Steer Markovian Agents under Model Uncertaintycited by 2