Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oracles
Not on the map. Why?
Abstract (arXiv)
For solving zero-sum games involving non-transitivity, a useful approach is to maintain a policy population to approximate the Nash Equilibrium (NE). Previous studies have shown that the Policy Space Response Oracles (PSRO) algorithm is an effective framework for solving such games. However, current methods initialize a new policy from scratch or inherit a single historical policy in Best Response (BR), missing the opportunity to leverage past policies to generate a better BR. In this paper, we propose Fusion-PSRO, which employs Nash Policy Fusion to initialize a new policy for BR training. Nash Policy Fusion serves as an implicit guiding policy that starts exploration on the current Meta-NE, thus providing a closer approximation to BR. Moreover, it insightfully captures a weighted moving average of past policies, dynamically adjusting these weights based on the Meta-NE in each iteration. This cumulative process further enhances the policy population. Empirical results on classic benchmarks show that Fusion-PSRO achieves lower exploitability, thereby mitigating the shortcomings of previous research on policy initialization in BR.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 31 May 2024.
Builds on
24 of 37 references.
References · 24
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2021Merging Models with Fisher-Weighted Averagingcited by 621
- 2016Deep Exploration via Bootstrapped DQNcited by 527
- 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
- 2023Nash Learning from Human Feedbackcited by 224
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
- 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 96
- 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
- 2023Deep Model Fusion: A Surveycited by 91
12 more
- 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
- 2020Real World Games Look Like Spinning Topscited by 66
- 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 51
- 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
- 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
- 2023PopulAtion Parameter Averaging (PAPA)cited by 23
- 2023Ensemble Reinforcement Learning: A Surveycited by 16
- 2023Policy Space Diversity for Non-Transitive Gamescited by 15
- 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
- 2022PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learningcited by 11
- 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
- 2022Efficient Policy Space Response Oraclescited by 8
Built on by
Citing papers per year: 2025: 2.
2 papers.