Self-adaptive PSRO: Towards an Automatic Population-based Game Solver
Not on the map. Why?
Abstract (arXiv)
Policy-Space Response Oracles (PSRO) as a general algorithmic framework has achieved state-of-the-art performance in learning equilibrium policies of two-player zero-sum games. However, the hand-crafted hyperparameter value selection in most of the existing works requires extensive domain knowledge, forming the main barrier to applying PSRO to different games. In this work, we make the first attempt to investigate the possibility of self-adaptively determining the optimal hyperparameter values in the PSRO framework. Our contributions are three-fold: (1) Using several hyperparameters, we propose a parametric PSRO that unifies the gradient descent ascent (GDA) and different PSRO variants. (2) We propose the self-adaptive PSRO (SPSRO) by casting the hyperparameter value selection of the parametric PSRO as a hyperparameter optimization (HPO) problem where our objective is to learn an HPO policy that can self-adaptively determine the optimal hyperparameter values during the running of the parametric PSRO. (3) To overcome the poor performance of online HPO methods, we propose a novel offline HPO approach to optimize the HPO policy based on the Transformer architecture. Experiments on various two-player zero-sum games demonstrate the superiority of SPSRO over different baselines.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 17 Apr 2024.
Builds on
40 of 62 references.
References · 40
- 2017Attention Is All You Needcited by 60,361
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2016Layer Normalizationcited by 5,669
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2023A Survey of Large Language Modelscited by 2,576
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2020Point Transformercited by 1,470
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
28 more
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
- 2016Learning to reinforcement learncited by 377
- 2021Transformers Can Do Bayesian Inferencecited by 360
- 2019Deep Learning for Symbolic Mathematicscited by 270
- 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
- 2022Multi-Game Decision Transformerscited by 214
- 2022Online Decision Transformercited by 196
- 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
- 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
- 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
- 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
- 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
- 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
- 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
- 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
- 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
- 2019A Generalized Training Approach for Multiagent Learningcited by 54
- 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 51
- 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
- 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
- 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
- 2018Deep Reinforcement Learning for Green Security Games with Real-Time Informationcited by 19
- 2021Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSROcited by 14
- 2021Iterative Empirical Game Solving via Single Policy Best Responsecited by 12
- 2019αα-Rank: Practically Scaling α-Rank through Stochastic Optimisationcited by 10
- 2021Evaluating Strategy Exploration in Empirical Game-Theoretic Analysiscited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.