# MATRIX: Multi-Agent Trajectory Generation with Diverse Contexts

arXiv: https://arxiv.org/abs/2403.06041 · HTML: https://mlsanity.com/paper/2403.06041/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 60 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 30 of 44)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs](https://mlsanity.com/paper/2402.07872/) (2024)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Leapfrog Diffusion Model for Stochastic Trajectory Prediction](https://mlsanity.com/paper/2303.10895/) (2023)
- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [Diverse Trajectory Forecasting with Determinantal Point Processes](https://mlsanity.com/paper/1907.04967/) (2019)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning](https://mlsanity.com/paper/2206.13387/) (2022)
- [LOKI: Long Term and Key Intentions for Trajectory Prediction](https://mlsanity.com/paper/2108.08236/) (2021)
- [Relational Graph Learning for Crowd Navigation](https://mlsanity.com/paper/1909.13165/) (2019)
- [Non-Probability Sampling Network for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2203.13471/) (2022)
- [Human Trajectory Prediction via Counterfactual Analysis](https://mlsanity.com/paper/2107.14202/) (2021)
- [RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting](https://mlsanity.com/paper/2108.01316/) (2021)

## Built on by (most cited first, 9 of 9)

- [Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs](https://mlsanity.com/paper/2407.07775/) (2024)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [Physical Plausibility-aware Trajectory Prediction via Locomotion Embodiment](https://mlsanity.com/paper/2503.17267/) (2025)
- [Towards Generalizable Safety in Crowd Navigation via Conformal Uncertainty Handling](https://mlsanity.com/paper/2508.05634/) (2025)
- [TrajEvo: Trajectory Prediction Heuristics Design via LLM-driven Evolution](https://mlsanity.com/paper/2508.05616/) (2025)
- [Annealed Winner-Takes-All for Motion Forecasting](https://mlsanity.com/paper/2409.11172/) (2024)
- [Recent Advances in Multi-Agent Human Trajectory Prediction: A Comprehensive Review](https://mlsanity.com/paper/2506.14831/) (2025)
- [Learning-Accelerated Optimization-based Trajectory Planning for Cooperative Aerial-Ground Handover Missions](https://mlsanity.com/paper/2605.19562/) (2026)

## Abstract (arXiv)

Data-driven methods have great advantages in modeling complicated human behavioral dynamics and dealing with many human-robot interaction applications. However, collecting massive and annotated real-world human datasets has been a laborious task, especially for highly interactive scenarios. On the other hand, algorithmic data generation methods are usually limited by their model capacities, making them unable to offer realistic and diverse data needed by various application users. In this work, we study trajectory-level data generation for multi-human or human-robot interaction scenarios and propose a learning-based automatic trajectory generation model, which we call Multi-Agent TRajectory generation with dIverse conteXts (MATRIX). MATRIX is capable of generating interactive human behaviors in realistic diverse contexts. We achieve this goal by modeling the explicit and interpretable objectives so that MATRIX can generate human motions based on diverse destinations and heterogeneous behaviors. We carried out extensive comparison and ablation studies to illustrate the effectiveness of our approach across various metrics. We also presented experiments that demonstrate the capability of MATRIX to serve as data augmentation for imitation-based motion planning.