mlsanityData to 14 Oct 2026

Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers

arXiv 2504.04395 ↗First posted 6 Apr 2025cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Competitive Pok'emon Singles (CPS) is a popular strategy game where players learn to exploit their opponent based on imperfect information in battles that can last more than one hundred stochastic turns. AI research in CPS has been led by heuristic tree search and online self-play, but the game may also create a platform to study adaptive policies trained offline on large datasets. We develop a pipeline to reconstruct the first-person perspective of an agent from logs saved from the third-person perspective of a spectator, thereby unlocking a dataset of real human battles spanning more than a decade that grows larger every day. This dataset enables a black-box approach where we train large sequence models to adapt to their opponent based solely on their input trajectory while selecting moves without explicit search of any kind. We study a progression from imitation learning to offline RL and offline fine-tuning on self-play data in the hardcore competitive setting of Pok'emon's four oldest (and most partially observed) game generations. The resulting agents outperform a recent LLM Agent approach and a strong heuristic search engine. While playing anonymously in online battles against humans, our best agents climb to rankings inside the top 10% of active players. All agent checkpoints, training details, datasets, and baselines are available at https://metamon.tech.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 78 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

49 of 78 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Layer Normalizationcited by 5,669
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
37 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2022A Generalist Agentcited by 808
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  7. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  8. 2016Learning to reinforcement learncited by 377
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  10. 2020Critic Regularized Regressioncited by 275
  11. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  13. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  14. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  15. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  16. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  17. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 114
  18. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  19. 2019Meta reinforcement learning as task inferencecited by 77
  20. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  21. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  22. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  23. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  24. 2021Deep Interactive Bayesian Reinforcement Learning via Meta-Learningcited by 29
  25. 2024PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Modelscited by 26
  26. 2021Student of Games: A unified learning algorithm for both perfect and imperfect information gamescited by 23
  27. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 21
  28. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 17
  29. 2025Pok'eChamp: an Expert-level Minimax Language Agentcited by 16
  30. 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Datasetcited by 15
  31. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  32. 2023Mastering Stacking of Diverse Shapes with Large-Scale Iterative Reinforcement Learning on Real Robotscited by 13
  33. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  34. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10
  35. 2021Search in Imperfect Information Gamescited by 8
  36. 2022Teamwork under extreme uncertainty: AI for Pokemon ranks 33rd in the worldcited by 1
  37. 2024Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Searchcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  2. 2026Automatic Generation of High-Performance RL Environmentscited by 0
  3. 2026PokaiTrainer: Scaling Equilibrium Search to Competitive Pok'emon VGCcited by 0
  4. 2026Porimon: An LLM-Based Pokémon Battle Agent Enhanced by Long/Short-Term Knowledge Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VGC-Bench: Towards Mastering Diverse Team Strategies in Competitive Pok'emon
  2. 2023Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games
  3. 2026Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
  4. 2025Large Language Models as Pok'emon Battle Agents: Strategic Play and Content Generation
  5. 2026Play Like Champions: Counterfactual Feedback Generation in Latent Space
  6. 2024Offline Fictitious Self-Play for Competitive Games
  7. 2026Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games
  8. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning