mlsanityData to 7 Oct 2026

Offline Fictitious Self-Play for Competitive Games

arXiv 2403.00841 ↗First posted 29 Feb 2024cs.MA cs.AI cs.GTcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) enables policy improvement from fixed datasets without online interactions, making it highly suitable for real-world applications lacking efficient simulators. Despite its success in the single-agent setting, offline multi-agent RL remains a challenge, especially in competitive games. Firstly, unaware of the game structure, it is impossible to interact with the opponents and conduct a major learning paradigm, self-play, for competitive games. Secondly, real-world datasets cannot cover all the state and action space in the game, resulting in barriers to identifying Nash equilibrium (NE). To address these issues, this paper introduces OFF-FSP, the first practical model-free offline RL algorithm for competitive games. We start by simulating interactions with various opponents by adjusting the weights of the fixed dataset with importance sampling. This technique allows us to learn the best responses to different opponents and employ the Offline Self-Play learning framework. To overcome the challenge of partial coverage, we combine the single-agent offline RL method with Fictitious Self-Play (FSP) to approximate NE by constraining the approximate best responses away from out-of-distribution actions. Experiments on matrix games, extensive-form poker, and board games demonstrate that OFF-FSP achieves significantly lower exploitability than state-of-the-art baselines. Finally, we validate OFF-FSP on a real-world human-robot competitive task, demonstrating its potential for solving complex, hard-to-simulate real-world problems.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 43 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

23 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning7 Aug
  3. 2022Offline Equilibrium Finding12 Jul
  4. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus1 Jun
  5. 2020Critic Regularized Regression26 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 16

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  3. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  5. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  8. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  9. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  10. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  11. 2020Improving Dialog Systems for Negotiation with Personality Modelingcited by 37
  12. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
4 more
  1. 2022Cooperative Multi-Agent Deep Reinforcement Learning for Reliable Surveillance via Autonomous Multi-UAV Controlcited by 25
  2. 2020Approximate exploitability: Learning a best response in large gamescited by 23
  3. 2023Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Explorationcited by 17
  4. 2022PerfectDou: Dominating DouDizhu with Perfect Information Distillationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Learning in Markov Games with General Function Approximation
  2. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?
  3. 2025DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play
  4. 2024Scalable Offline Reinforcement Learning for Mean Field Games
  5. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
  6. 2019An Optimistic Perspective on Offline Reinforcement Learning
  7. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  8. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games