mlsanityData to 14 Oct 2026

A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games

arXiv 2206.05825 ↗First posted 12 Jun 2022cs.LG cs.AI cs.GTcited by 66

Not on the map. Why?

Abstract (arXiv)

This work studies an algorithm, which we call magnetic mirror descent, that is inspired by mirror descent and the non-Euclidean proximal gradient algorithm. Our contribution is demonstrating the virtues of magnetic mirror descent as both an equilibrium solver and as an approach to reinforcement learning in two-player zero-sum games. These virtues include: 1) Being the first quantal response equilibria solver to achieve linear convergence for extensive-form games with first order feedback; 2) Being the first standard reinforcement learning algorithm to achieve empirically competitive results with CFR in tabular settings; 3) Achieving favorable performance in 3x3 Dark Hex and Phantom Tic-Tac-Toe as a self-play deep reinforcement learning algorithm.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 33 of its 88 references that are in the map. First posted on arXiv 12 Jun 2022.

Builds on

33 of 88 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2014Convex Optimization: Algorithms and Complexitycited by 575
  3. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  5. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  6. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  7. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  8. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  9. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  10. 2020Mirror Descent Policy Optimizationcited by 106
  11. 2018Deep Counterfactual Regret Minimizationcited by 103
  12. 2014Solving Large Imperfect Information Games Using CFR+cited by 78
21 more
  1. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  2. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  3. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  4. 2019Differentiable Game Mechanicscited by 43
  5. 2017Catalyst Acceleration for First-order Convex Optimization: from Theory to Practicecited by 39
  6. 2021Extragradient Method: O(1/K) Last-Iterate Convergence for Monotone Variational Inequalities and Connections With Cocoercivitycited by 39
  7. 2019No Press Diplomacy: Modeling Multi-Agent Gameplaycited by 38
  8. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  9. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  10. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  11. 2021No-Press Diplomacy from Scratchcited by 31
  12. 2018Variance Reduction in Monte Carlo Counterfactual Regret Minimization (VR-MCCFR) for Extensive Form Games using Baselinescited by 25
  13. 2018What game are we playing? End-to-end learning in normal and extensive form gamescited by 22
  14. 2018Online Convex Optimization for Sequential Decision Processes and Extensive-Form Gamescited by 21
  15. 2018Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimizationcited by 20
  16. 2020The Advantage Regret-Matching Actor-Criticcited by 17
  17. 2018Double Neural Counterfactual Regret Minimizationcited by 16
  18. 2022Generalized Data Distribution Iterationcited by 12
  19. 2019Low-Variance and Zero-Variance Baselines for Extensive-Form Gamescited by 9
  20. 2019Large Scale Learning of Agent Rationality in Two-Player Zero-Sum Gamescited by 6
  21. 2022Equilibrium Finding in Normal-Form Games Via Greedy Regret Minimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 17; 2024: 12; 2025: 14; 2026: 19.

66 papers.

Sort:

By area:

  1. 2023Nash Learning from Human Feedbackcited by 224
  2. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  3. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
  4. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  5. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 18
  6. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 16
  7. 2023Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditionscited by 14
  8. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  9. 2022Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attackscited by 12
  10. 2022Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Gamescited by 11
  11. 2023Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problemscited by 11
  12. 2023Mastering Strategy Card Game (Hearthstone) with Improved Techniquescited by 10
38 more
  1. 2023Multi-Player Zero-Sum Markov Games with Networked Separable Interactionscited by 10
  2. 2024Large Language Models as Agents in Two-Player Gamescited by 10
  3. 2023Adaptively Perturbed Mirror Descent for Learning in Gamescited by 9
  4. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  5. 2022Symmetric (Optimistic) Natural Policy Gradient for Multi-agent Learning with Parameter Convergencecited by 8
  6. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
  7. 2025Scalable Decision Making for Games of Imperfect Informationcited by 8
  8. 2023Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RLcited by 7
  9. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  10. 2025Multiplayer Nash Preference Optimizationcited by 7
  11. 2023Abstracting Imperfect Information Away from Two-Player Zero-Sum Gamescited by 6
  12. 2023Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Gamescited by 6
  13. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  14. 2024A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergencecited by 5
  15. 2025Multiplayer Federated Learning: Reaching Equilibrium with Less Communicationcited by 5
  16. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  17. 2023Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updatescited by 4
  18. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 4
  19. 2023The Update-Equivalence Framework for Decision-Time Planningcited by 3
  20. 2023Smooth Nash Equilibria: Algorithms and Complexitycited by 3
  21. 2024LiteEFG: An Efficient Python Library for Solving Extensive-form Gamescited by 3
  22. 2025Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Gamescited by 3
  23. 2025Extragradient Method for (L0, L1)-Lipschitz Root-finding Problemscited by 3
  24. 2023Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Controlcited by 2
  25. 2024Finite-Sample Guarantees for Learning Dynamics in Zero-Sum Polymatrix Gamescited by 2
  26. 2025Modelling bounded rational decision-making through Wasserstein constraintscited by 2
  27. 2025Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economicscited by 2
  28. 2025NashPG: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibriacited by 2
  29. 2026Which Nash Equilibrium? Solver-Dependent Selection on Zero-Sum Nash Polytopescited by 2
  30. 2024Configurable Mirror Descent: Towards a Unification of Decision Makingcited by 1
  31. 2024TrafficGamer: Reliable and Flexible Traffic Simulation for Safety-Critical Scenarios with Game-Theoretic Oraclescited by 1
  32. 2025Convergence and Connectivity: Dynamics of Multi-Agent Q-Learning in Random Networkscited by 1
  33. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Gamescited by 1
  34. 2026Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Gamescited by 1
  35. 2026EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Gamescited by 1
  36. 2026How Much Due Diligence Before You Bid? Learning in Intractable Takeover Auctionscited by 1
  37. 2026The Curvature Shadow: An Apparent Failure of Maximum-Entropy Equilibrium Selection is a Removable Artifactcited by 1
  38. 2023Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbationscited by 0

50 most cited of 66 All 66 (JSON)