# Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach

arXiv: https://arxiv.org/abs/2008.03880 · HTML: https://mlsanity.com/paper/2008.03880/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2020.

## Built on by (most cited first, 30 of 47)

- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [UnO: Unsupervised Occupancy Fields for Perception and Forecasting](https://mlsanity.com/paper/2406.08691/) (2024)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving](https://mlsanity.com/paper/2308.01471/) (2023)
- [Robots That Can See: Leveraging Human Pose for Trajectory Prediction](https://mlsanity.com/paper/2309.17209/) (2023)
- [EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction](https://mlsanity.com/paper/2308.06564/) (2023)
- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [CVAE-H: Conditionalizing Variational Autoencoders via Hypernetworks and Trajectory Forecasting for Autonomous Driving](https://mlsanity.com/paper/2201.09874/) (2022)
- [Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling](https://mlsanity.com/paper/2112.00298/) (2021)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [Learning to Navigate Intersections with Unsupervised Driver Trait Inference](https://mlsanity.com/paper/2109.06783/) (2021)
- [Vision-based Multi-future Trajectory Prediction: A Survey](https://mlsanity.com/paper/2302.10463/) (2023)
- [SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2105.04642/) (2021)
- [CTRMs: Learning to Construct Cooperative Timed Roadmaps for Multi-agent Path Planning in Continuous Spaces](https://mlsanity.com/paper/2201.09467/) (2022)
- [Conditional Unscented Autoencoders for Trajectory Prediction](https://mlsanity.com/paper/2310.19944/) (2023)
- [GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction](https://mlsanity.com/paper/2506.21121/) (2025)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [A Game-Theoretic Framework for Joint Forecasting and Planning](https://mlsanity.com/paper/2308.06137/) (2023)
- [Unified Human Localization and Trajectory Prediction with Monocular Vision](https://mlsanity.com/paper/2503.03535/) (2025)
- [Motion Forecasting via Model-Based Risk Minimization](https://mlsanity.com/paper/2409.10585/) (2024)
- [Harmonious Group Choreography with Trajectory-Controllable Diffusion](https://mlsanity.com/paper/2403.06189/) (2024)
- [Conformal Trajectory Prediction with Multi-View Data Integration in Cooperative Driving](https://mlsanity.com/paper/2408.00374/) (2024)
- [Learning responsibility allocations for multi-agent interactions: A differentiable optimization approach with control barrier functions](https://mlsanity.com/paper/2410.07409/) (2024)
- [Bayesian Multiobject Tracking With Neural-Enhanced Motion and Measurement Models](https://mlsanity.com/paper/2506.18124/) (2025)
- [OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction](https://mlsanity.com/paper/2507.23657/) (2025)
- [Conditional Generative Adversarial Networks for Speed Control in Trajectory Simulation](https://mlsanity.com/paper/2103.11471/) (2021)
- [Unified Control Framework for Real-Time Interception and Obstacle Avoidance of Fast-Moving Objects with Diffusion Variational Autoencoder](https://mlsanity.com/paper/2209.13628/) (2022)
- [Data-driven Probabilistic Trajectory Learning with High Temporal Resolution in Terminal Airspace](https://mlsanity.com/paper/2409.17359/) (2024)

## Abstract (arXiv)

Human behavior prediction models enable robots to anticipate how humans may react to their actions, and hence are instrumental to devising safe and proactive robot planning algorithms. However, modeling complex interaction dynamics and capturing the possibility of many possible outcomes in such interactive settings is very challenging, which has recently prompted the study of several different approaches. In this work, we provide a self-contained tutorial on a conditional variational autoencoder (CVAE) approach to human behavior prediction which, at its core, can produce a multimodal probability distribution over future human trajectories conditioned on past interactions and candidate robot future actions. Specifically, the goals of this tutorial paper are to review and build a taxonomy of state-of-the-art methods in human behavior prediction, from physics-based to purely data-driven methods, provide a rigorous yet easily accessible description of a data-driven, CVAE-based approach, highlight important design characteristics that make this an attractive model to use in the context of model-based planning for human-robot interactions, and provide important design considerations when using this class of models.