# TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

arXiv: https://arxiv.org/abs/2110.04507 · HTML: https://mlsanity.com/paper/2110.04507/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00714/) (2021)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.13888/) (2020)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [BeBold: Exploration Beyond the Boundary of Explored Regions](https://mlsanity.com/paper/2012.08621/) (2020)

## Built on by (most cited first, 18 of 18)

- [Synthesizing Physical Character-Scene Interactions](https://mlsanity.com/paper/2302.00883/) (2023)
- [JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2112.04907/) (2021)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization](https://mlsanity.com/paper/2207.05631/) (2022)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Games](https://mlsanity.com/paper/2310.03354/) (2023)
- [An Empirical Study on Google Research Football Multi-agent Scenarios](https://mlsanity.com/paper/2305.09458/) (2023)
- [Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03429/) (2023)
- [Multi-Agent Diagnostics for Robustness via Illuminated Diversity](https://mlsanity.com/paper/2401.13460/) (2024)
- [Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs](https://mlsanity.com/paper/2306.15903/) (2023)
- [Robust Agents in Open-Ended Worlds](https://mlsanity.com/paper/2512.08139/) (2025)
- [Learning cooperative behaviours in adversarial multi-agent systems](https://mlsanity.com/paper/2302.05528/) (2023)
- [Diverse Policies Converge in Reward-free Markov Decision Processe](https://mlsanity.com/paper/2308.11924/) (2023)
- [ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2606.24601/) (2026)
- [HySTAR: Anchored Hypergraphs for Stable Credit Assignment in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.31531/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has achieved super-human performance on complex video games (e.g., StarCraft II and Dota II). However, current DRL systems still suffer from challenges of multi-agent coordination, sparse rewards, stochastic environments, etc. In seeking to address these challenges, we employ a football video game, e.g., Google Research Football (GRF), as our testbed and develop an end-to-end learning-based AI system (denoted as TiKick) to complete this challenging task. In this work, we first generated a large replay dataset from the self-playing of single-agent experts, which are obtained from league training. We then developed a distributed learning system and new offline algorithms to learn a powerful multi-agent AI from the fixed single-agent dataset. To the best of our knowledge, Tikick is the first learning-based AI system that can take over the multi-agent Google Research Football full game, while previous work could either control a single agent or experiment on toy academic scenarios. Extensive experiments further show that our pre-trained model can accelerate the training process of the modern multi-agent algorithm and our method achieves state-of-the-art performances on various academic scenarios.