mlsanityData to 7 Oct 2026

Offline Equilibrium Finding

arXiv 2207.05285 ↗First posted 12 Jul 2022cs.AI cs.GT cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (offline RL) is an emerging field that has recently begun gaining attention across various application domains due to its ability to learn strategies from earlier collected datasets. Offline RL proved very successful, paving a path to solving previously intractable real-world problems, and we aim to generalize this paradigm to a multiplayer-game setting. To this end, we introduce a problem of offline equilibrium finding (OEF) and construct multiple types of datasets across a wide range of games using several established methods. To solve the OEF problem, we design a model-based framework that can directly apply any online equilibrium finding algorithm to the OEF setting while making minimal changes. The three most prominent contemporary online equilibrium finding algorithms are adapted to the context of OEF, creating three model-based variants: OEF-PSRO and OEF-CFR, which generalize the widely-used algorithms PSRO and Deep CFR to compute Nash equilibria (NEs), and OEF-JPSRO, which generalizes the JPSRO to calculate (Coarse) Correlated equilibria ((C)CEs). We also combine the behavior cloning policy with the model-based policy to further improve the performance and provide a theoretical guarantee of the solution quality. Extensive experimental results demonstrate the superiority of our approach over offline RL algorithms and the importance of using model-based methods for OEF problems. We hope our work will contribute to advancing research in large-scale equilibrium finding.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 74 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

41 of 74 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  6. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  7. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  8. 2017Learning with Opponent-Learning Awarenesscited by 218
  9. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  10. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  11. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  12. 2018Deep Counterfactual Regret Minimizationcited by 102
19 more
  1. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  2. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  3. 2018Solving Imperfect-Information Games via Discounted Regret Minimizationcited by 68
  4. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  5. 2018Learning Policy Representations in Multiagent Systemscited by 48
  6. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  7. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  8. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 40
  9. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  10. 2020Game Plan: What AI can do for Football, and What Football can do for AIcited by 27
  11. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  12. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  13. 2017A Deep Policy Inference Q-Network for Multi-Agent Systemscited by 24
  14. 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
  15. 2021Model-Based Opponent Modelingcited by 21
  16. 2019Single Deep Counterfactual Regret Minimizationcited by 20
  17. 2018Double Neural Counterfactual Regret Minimizationcited by 16
  18. 2018A Generalised Method for Empirical Game Theoretic Analysiscited by 15
  19. 2021CFR-MIX: Solving Imperfect Information Extensive-Form Games with Combinatorial Action Spacecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Offline Fictitious Self-Play for Competitive Gamescited by 1
  2. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Learning in Markov Games with General Function Approximation
  2. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  3. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2024Scalable Offline Reinforcement Learning for Mean Field Games
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2022Leveraging Offline Data in Online Reinforcement Learning
  8. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularization