mlsanityData to 14 Oct 2026

Smoothing Policy Iteration for Zero-sum Markov Games

arXiv 2212.01623 ↗First posted 3 Dec 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Zero-sum Markov Games (MGs) has been an efficient framework for multi-agent systems and robust control, wherein a minimax problem is constructed to solve the equilibrium policies. At present, this formulation is well studied under tabular settings wherein the maximum operator is primarily and exactly solved to calculate the worst-case value function. However, it is non-trivial to extend such methods to handle complex tasks, as finding the maximum over large-scale action spaces is usually cumbersome. In this paper, we propose the smoothing policy iteration (SPI) algorithm to solve the zero-sum MGs approximately, where the maximum operator is replaced by the weighted LogSumExp (WLSE) function to obtain the nearly optimal equilibrium policies. Specially, the adversarial policy is served as the weight function to enable an efficient sampling over action spaces.We also prove the convergence of SPI and analyze its approximation error in ∞ -norm based on the contraction mapping theorem. Besides, we propose a model-based algorithm called Smooth adversarial Actor-critic (SaAC) by extending SPI with the function approximations. The target value related to WLSE function is evaluated by the sampled trajectories and then mean square error is constructed to optimize the value function, and the gradient-ascent-descent methods are adopted to optimize the protagonist and adversarial policies jointly. In addition, we incorporate the reparameterization technique in model-based gradient back-propagation to prevent the gradient vanishing due to sampling from the stochastic policies. We verify our algorithm in both tabular and function approximation settings. Results show that SPI can approximate the worst-case value function with a high accuracy and SaAC can stabilize the training process and improve the adversarial robustness in a large margin.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 3 Dec 2022.

Builds on

16 of 31 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Robust Adversarial Reinforcement Learningcited by 451
  3. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  4. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  5. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 92
  6. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  8. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  9. 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 57
  10. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 43
  11. 2019Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learningcited by 23
  12. 2020Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Criticcited by 12
4 more
  1. 2019A Generalized Minimax Q-learning Algorithm for Two-Player Zero-Sum Stochastic Gamescited by 8
  2. 2021Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and modelcited by 8
  3. 2022Improve Generalization of Driving Policy at Signalized Intersections with Adversarial Learningcited by 2
  4. 2020Reinforcement Solver for H-infinity Filter with Bounded Noisecited by 1

Built on by

Not cited yet.