# trajdata: A Unified Interface to Multiple Human Trajectory Datasets

arXiv: https://arxiv.org/abs/2307.13924 · HTML: https://mlsanity.com/paper/2307.13924/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion](https://mlsanity.com/paper/2304.01893/) (2023)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [BITS: Bi-level Imitation for Traffic Simulation](https://mlsanity.com/paper/2208.12403/) (2022)
- [Identifying Driver Interactions via Conditional Behavior Prediction](https://mlsanity.com/paper/2104.09959/) (2021)
- [Tree-structured Policy Planning with Learned Behavior Models](https://mlsanity.com/paper/2301.11902/) (2023)
- [Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning](https://mlsanity.com/paper/2209.11820/) (2022)
- [OpenTraj: Assessing Prediction Complexity in Human Trajectories Datasets](https://mlsanity.com/paper/2010.00890/) (2020)
- [Uncertainty estimation for Cross-dataset performance in Trajectory prediction](https://mlsanity.com/paper/2205.07310/) (2022)
- [The Atlas Benchmark: an Automated Evaluation Framework for Human Motion Prediction](https://mlsanity.com/paper/2207.09830/) (2022)
- [Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models](https://mlsanity.com/paper/2210.14584/) (2022)

## Built on by (most cited first, 25 of 25)

- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios](https://mlsanity.com/paper/2312.13303/) (2023)
- [SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries](https://mlsanity.com/paper/2401.00391/) (2023)
- [Producing and Leveraging Online Map Uncertainty in Trajectory Prediction](https://mlsanity.com/paper/2403.16439/) (2024)
- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory](https://mlsanity.com/paper/2403.10052/) (2024)
- [Data Scaling Laws for End-to-End Autonomous Driving](https://mlsanity.com/paper/2504.04338/) (2025)
- [LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation](https://mlsanity.com/paper/2504.11521/) (2025)
- [Accelerating Online Mapping and Behavior Prediction via Direct BEV Feature Attention](https://mlsanity.com/paper/2407.06683/) (2024)
- [Prediction Horizon Requirements for Automated Driving: Optimizing Safety, Comfort, and Efficiency](https://mlsanity.com/paper/2402.03893/) (2024)
- [OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction](https://mlsanity.com/paper/2507.23657/) (2025)
- [Implicit Maximum Likelihood Estimation for Real-time Generative Model Predictive Control](https://mlsanity.com/paper/2603.13733/) (2026)
- [123D: Unifying Multi-Modal Autonomous Driving Data at Scale](https://mlsanity.com/paper/2605.08084/) (2026)
- [Toward Unified Practices in Trajectory Prediction Research on Bird's-Eye-View Datasets](https://mlsanity.com/paper/2405.00604/) (2024)
- [STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models](https://mlsanity.com/paper/2509.14801/) (2025)
- [DeepUrban: Interaction-Aware Trajectory Prediction and Planning for Automated Driving by Aerial Imagery](https://mlsanity.com/paper/2601.10554/) (2026)
- [Poly-Autoregressive Prediction for Modeling Interactions](https://mlsanity.com/paper/2502.08646/) (2025)
- [HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic](https://mlsanity.com/paper/2602.03447/) (2026)
- [Amelia: A Large Dataset and Benchmark for Airport Surface Movement Forecasting](https://mlsanity.com/paper/2407.21185/) (2024)
- [MetaDAT: Generalizable Trajectory Prediction via Meta Pre-training and Data-Adaptive Test-Time Updating](https://mlsanity.com/paper/2603.09419/) (2026)
- [Uncertainty-Aware Trajectory Prediction: A Unified Framework Harnessing Positional and Semantic Uncertainties](https://mlsanity.com/paper/2603.29362/) (2026)
- [StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasets](https://mlsanity.com/paper/2606.04271/) (2026)
- [Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings](https://mlsanity.com/paper/2606.30777/) (2026)
- [End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation](https://mlsanity.com/paper/2607.18637/) (2026)
- [STRIDE: Automated Evaluation of Text-to-Trajectory Alignment across Diverse Contexts](https://mlsanity.com/paper/2609.34799/) (2026)

## Abstract (arXiv)

The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, real-world human trajectory datasets for autonomous vehicles (AVs) and pedestrian motion tracking. While such datasets have been a boon for the community, they each use custom and unique data formats and APIs, making it cumbersome for researchers to train and evaluate methods across multiple datasets. To remedy this, we present trajdata: a unified interface to multiple human trajectory datasets. At its core, trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data. As a demonstration of its capabilities, in this work we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a rich understanding of the data underpinning much of current pedestrian and AV motion forecasting research, and proposing suggestions for future datasets from these insights. trajdata is permissively licensed (Apache 2.0) and can be accessed online at https://github.com/NVlabs/trajdata