# Stochastic Prediction of Multi-Agent Interactions from Partial Observations

arXiv: https://arxiv.org/abs/1902.09641 · HTML: https://mlsanity.com/paper/1902.09641/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 26 of 26)

- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [Spatio-Temporal Dynamic Inference Network for Group Activity Recognition](https://mlsanity.com/paper/2108.11743/) (2021)
- [How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting](https://mlsanity.com/paper/2203.04781/) (2022)
- [CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning](https://mlsanity.com/paper/2106.13364/) (2021)
- [Neural Relational Inference with Fast Modular Meta-learning](https://mlsanity.com/paper/2310.07015/) (2023)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [Learning Temporal Dynamics from Cycles in Narrated Video](https://mlsanity.com/paper/2101.02337/) (2021)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports](https://mlsanity.com/paper/2410.17785/) (2024)
- [Aware of the History: Trajectory Forecasting with the Local Behavior Data](https://mlsanity.com/paper/2207.09646/) (2022)
- [Inferring Player Location in Sports Matches: Multi-Agent Spatial Imputation from Limited Observations](https://mlsanity.com/paper/2302.06569/) (2023)
- [FootBots: A Transformer-based Architecture for Motion Prediction in Soccer](https://mlsanity.com/paper/2406.19852/) (2024)
- [Agent-Centric Representations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.09402/) (2021)
- [Group Activity Recognition in Basketball Tracking Data -- Neural Embeddings in Team Sports (NETS)](https://mlsanity.com/paper/2209.00451/) (2022)
- [Representing Spatial Trajectories as Distributions](https://mlsanity.com/paper/2210.01322/) (2022)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention](https://mlsanity.com/paper/2211.06027/) (2022)
- [Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2605.10717/) (2026)
- [About latent roles in forecasting players in team sports](https://mlsanity.com/paper/2304.08272/) (2023)
- [Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes](https://mlsanity.com/paper/2405.20743/) (2024)
- [Modeling Human Behavior Part I -- Learning and Belief Approaches](https://mlsanity.com/paper/2205.06485/) (2022)
- [Rethinking Trajectory Prediction via "Team Game"](https://mlsanity.com/paper/2210.08793/) (2022)

## Abstract (arXiv)

We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.