Configurable Mirror Descent: Towards a Unification of Decision Making
Not on the map. Why?
Abstract (arXiv)
Decision-making problems, categorized as single-agent, e.g., Atari, cooperative multi-agent, e.g., Hanabi, competitive multi-agent, e.g., Hold'em poker, and mixed cooperative and competitive, e.g., football, are ubiquitous in the real world. Various methods are proposed to address the specific decision-making problems. Despite the successes in specific categories, these methods typically evolve independently and cannot generalize to other categories. Therefore, a fundamental question for decision-making is: emphCan we develop a single algorithm to tackle ALL categories of decision-making problems? There are several main challenges to address this question: i) different decision-making categories involve different numbers of agents and different relationships between agents, ii) different categories have different solution concepts and evaluation measures, and iii) there lacks a comprehensive benchmark covering all the categories. This work presents a preliminary attempt to address the question with three main contributions. i) We propose the generalized mirror descent (GMD), a generalization of MD variants, which considers multiple historical policies and works with a broader class of Bregman divergences. ii) We propose the configurable mirror descent (CMD) where a meta-controller is introduced to dynamically adjust the hyper-parameters in GMD conditional on the evaluation measures. iii) We construct the textscGameBench with 15 academic-friendly games across different decision-making categories. Extensive experiments demonstrate that CMD achieves empirically competitive or better outcomes compared to baselines while providing the capability of exploring diverse dimensions of decision making.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 64 of its 98 references that are in the map. First posted on arXiv 20 May 2024.
Builds on
64 of 98 references.
References · 64
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021Convex optimizationcited by 2,515
- 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
- 2017Counterfactual Multi-Agent Policy Gradientscited by 751
- 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
- 2019The StarCraft Multi-Agent Challengecited by 597
- 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
- 2016Input Convex Neural Networkscited by 414
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
- 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
38 more
- 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
- 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
- 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
- 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
- 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
- 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
- 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
- 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
- 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
- 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
- 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
- 2020Mirror Descent Policy Optimizationcited by 106
- 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
- 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
- 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
- 2014Solving Large Imperfect Information Games Using CFR+cited by 78
- 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 68
- 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
- 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
- 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 54
- 2021Adaptive Learning in Continuous Games: Optimal Regret Bounds and Convergence to Nash Equilibriumcited by 51
- 2020Tight last-iterate convergence rates for no-regret learning in multi-player gamescited by 49
- 2019Finding Friend and Foe in Multi-Agent Gamescited by 42
- 2023ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agentscited by 33
- 2022On Last-Iterate Convergence Beyond Zero-Sum Gamescited by 31
- 2021Last-iterate Convergence in Extensive-Form Gamescited by 28
- 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
- 2021Reinforcement Learning for Quantitative Tradingcited by 24
- 2022Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variancecited by 24
- 2021Student of Games: A unified learning algorithm for both perfect and imperfect information gamescited by 23
- 2021Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recallcited by 21
- 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
- 2022The Power of Regularization in Solving Extensive-Form Gamescited by 21
- 2019Coarse Correlation in Extensive-Form Gamescited by 17
- 2022Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worldscited by 17
- 2021Solving Common-Payoff Games with Approximate Policy Iterationcited by 15
- 2019Fast and Furious Learning in Zero-Sum Games: Vanishing Regret with Non-Vanishing Step Sizescited by 14
- 2022Alternating Mirror Descent for Constrained Min-Max Gamescited by 13
50 most cited of 64 All 64 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.