# Generating Multi-Agent Trajectories using Programmatic Weak Supervision

arXiv: https://arxiv.org/abs/1803.07612 · HTML: https://mlsanity.com/paper/1803.07612/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2018.

## Built on by (most cited first, 30 of 46)

- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [Clifford Group Equivariant Simplicial Message Passing Networks](https://mlsanity.com/paper/2402.10011/) (2024)
- [Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction](https://mlsanity.com/paper/2206.01899/) (2022)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision](https://mlsanity.com/paper/2203.13270/) (2022)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling](https://mlsanity.com/paper/2102.03291/) (2021)
- [Creating Training Sets via Weak Indirect Supervision](https://mlsanity.com/paper/2110.03484/) (2021)
- [Interaction Modeling with Multiplex Attention](https://mlsanity.com/paper/2208.10660/) (2022)
- [Multi-Agent Imitation Learning with Copulas](https://mlsanity.com/paper/2107.04750/) (2021)
- [baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents](https://mlsanity.com/paper/2104.11980/) (2021)
- [Estimating counterfactual treatment outcomes over time in complex multiagent scenarios](https://mlsanity.com/paper/2206.01900/) (2022)
- [Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical Bi-LSTM](https://mlsanity.com/paper/2306.08206/) (2023)
- [TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports](https://mlsanity.com/paper/2410.17785/) (2024)
- [Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning](https://mlsanity.com/paper/2305.17886/) (2023)
- [Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations](https://mlsanity.com/paper/2305.13030/) (2023)
- [All You Need In Sign Language Production](https://mlsanity.com/paper/2201.01609/) (2022)
- [FootBots: A Transformer-based Architecture for Motion Prediction in Soccer](https://mlsanity.com/paper/2406.19852/) (2024)
- [Pedestrian Trajectory Prediction with Missing Data: Datasets, Imputation, and Benchmarking](https://mlsanity.com/paper/2411.00174/) (2024)
- [OpenSTARLab: Open Approach for Spatio-Temporal Agent Data Analysis in Soccer](https://mlsanity.com/paper/2502.02785/) (2025)
- [Data-driven Analysis for Understanding Team Sports Behaviors](https://mlsanity.com/paper/2102.07545/) (2021)
- [Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis](https://mlsanity.com/paper/2111.15186/) (2021)
- [Group Activity Recognition in Basketball Tracking Data -- Neural Embeddings in Team Sports (NETS)](https://mlsanity.com/paper/2209.00451/) (2022)
- [Mathematical models for off-ball scoring prediction in basketball](https://mlsanity.com/paper/2406.08749/) (2024)
- [JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation](https://mlsanity.com/paper/2509.22522/) (2025)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2605.10717/) (2026)
- [About latent roles in forecasting players in team sports](https://mlsanity.com/paper/2304.08272/) (2023)

## Abstract (arXiv)

We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way. We present a hierarchical framework that can effectively learn such sequential generative models. Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.