# Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations

arXiv: https://arxiv.org/abs/2305.13030 · HTML: https://mlsanity.com/paper/2305.13030/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 90 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction](https://mlsanity.com/paper/1703.01030/) (2017)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Cross-Domain Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2110.03684/) (2021)
- [Generating Multi-Agent Trajectories using Programmatic Weak Supervision](https://mlsanity.com/paper/1803.07612/) (2018)
- [Cross-domain Imitation from Observations](https://mlsanity.com/paper/2105.10037/) (2021)
- [Generating Long-term Trajectories Using Deep Hierarchical Networks](https://mlsanity.com/paper/1706.07138/) (2017)
- [Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluation](https://mlsanity.com/paper/1805.11088/) (2018)
- [GCS: Graph-based Coordination Strategy for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.06257/) (2022)
- [Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction](https://mlsanity.com/paper/2206.01899/) (2022)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Learn what matters: cross-domain imitation learning with task-relevant embeddings](https://mlsanity.com/paper/2209.12093/) (2022)
- [Multi-Agent Imitation Learning with Copulas](https://mlsanity.com/paper/2107.04750/) (2021)
- [Estimating counterfactual treatment outcomes over time in complex multiagent scenarios](https://mlsanity.com/paper/2206.01900/) (2022)
- [Imitation Learning from Observations under Transition Model Disparity](https://mlsanity.com/paper/2204.11446/) (2022)
- [Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations](https://mlsanity.com/paper/1801.10459/) (2018)

## Built on by (most cited first, 9 of 9)

- [Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning](https://mlsanity.com/paper/2305.17886/) (2023)
- [A Strategic Framework for Optimal Decisions in Football 1-vs-1 Shot-Taking Situations: An Integrated Approach of Machine Learning, Theory-Based Modeling, and Game Theory](https://mlsanity.com/paper/2307.14732/) (2023)
- [OpenSTARLab: Open Approach for Spatio-Temporal Agent Data Analysis in Soccer](https://mlsanity.com/paper/2502.02785/) (2025)
- [TrackID3x3: A Dataset and Algorithm for Multi-Player Tracking with Identification and Pose Estimation in 3x3 Basketball Full-court Videos](https://mlsanity.com/paper/2503.18282/) (2025)
- [Evaluating Movement Initiation Timing in Ultimate Frisbee via Temporal Counterfactuals](https://mlsanity.com/paper/2508.17611/) (2025)
- [Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning](https://mlsanity.com/paper/2510.10451/) (2025)
- [Monte Carlo Pass Search: Using Trajectory Generation for 3D Counterfactual Pass Evaluation in Football](https://mlsanity.com/paper/2606.11120/) (2026)
- [R2D-RL: A RoboCup 2D Soccer Environment for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.18786/) (2026)
- [Strategic Multi-Agent Learning for Interpretable Action Valuation of All Players in Football](https://mlsanity.com/paper/2610.05961/) (2026)

## Abstract (arXiv)

Modeling of real-world biological multi-agents is a fundamental problem in various scientific and engineering fields. Reinforcement learning (RL) is a powerful framework to generate flexible and diverse behaviors in cyberspace; however, when modeling real-world biological multi-agents, there is a domain gap between behaviors in the source (i.e., real-world data) and the target (i.e., cyberspace for RL), and the source environment parameters are usually unknown. In this paper, we propose a method for adaptive action supervision in RL from real-world demonstrations in multi-agent scenarios. We adopt an approach that combines RL and supervised learning by selecting actions of demonstrations in RL based on the minimum distance of dynamic time warping for utilizing the information of the unknown source dynamics. This approach can be easily applied to many existing neural network architectures and provide us with an RL model balanced between reproducibility as imitation and generalization ability to obtain rewards in cyberspace. In the experiments, using chase-and-escape and football tasks with the different dynamics between the unknown source and target environments, we show that our approach achieved a balance between the reproducibility and the generalization ability compared with the baselines. In particular, we used the tracking data of professional football players as expert demonstrations in football and show successful performances despite the larger gap between behaviors in the source and target environments than the chase-and-escape task.