mlsanityData to 7 Oct 2026

Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

arXiv 2104.10133 ↗First posted 20 Apr 2021cs.CV cs.LG cs.ROcited by 532

Not on the map. Why?

Cited by 138 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As autonomous driving systems mature, motion forecasting has received increasing attention as a critical requirement for planning. Of particular importance are interactive situations such as merges, unprotected turns, etc., where predicting individual object motion is not sufficient. Joint predictions of multiple objects are required for effective route planning. There has been a critical need for high-quality motion data that is rich in both interactions and annotation to develop motion planning models. In this work, we introduce the most diverse interactive motion dataset to our knowledge, and provide specific labels for interacting objects suitable for developing joint prediction models. With over 100,000 scenes, each 20 seconds long at 10 Hz, our new dataset contains more than 570 hours of unique data over 1750 km of roadways. It was collected by mining for interesting interactions between vehicles, pedestrians, and cyclists across six cities within the United States. We use a high-accuracy 3D auto-labeling system to generate high quality 3D bounding boxes for each road agent, and provide corresponding high definition 3D maps for each scene. Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models. Finally, we provide strong baseline models for individual-agent prediction and joint-prediction. We hope that this new large-scale interactive motion dataset will provide new opportunities for advancing motion forecasting models.

Key facts. Cited by 532 papers in the map (data through 7 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

27 of 41 references.

References · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  5. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  6. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  7. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  8. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  9. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  10. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  11. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  12. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
15 more
  1. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  2. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  3. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  4. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  5. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  6. 2019Multiple Futures Predictioncited by 178
  7. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  8. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  9. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  10. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  11. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  12. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  13. 2021Auto4D: Learning to Label 4D Objects from Sequential Point Cloudscited by 42
  14. 2019Analyzing the Variety Loss in the Context of Probabilistic Trajectory Predictioncited by 30
  15. 2020PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous drivingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 57; 2023: 100; 2024: 113; 2025: 124; 2026: 121.

532 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  5. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  6. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  7. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  8. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  9. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  10. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
  11. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  12. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
38 more
  1. 2022V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Drivingcited by 157
  2. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  3. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  4. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  5. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  6. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  7. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  9. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  10. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  11. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  12. 2023V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecastingcited by 114
  13. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  14. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
  15. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  16. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  17. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  18. 2023The Waymo Open Sim Agents Challengecited by 86
  19. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  20. 2024Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformerscited by 76
  21. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  22. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  23. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 72
  24. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  25. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  26. 2025Robust Autonomy Emerges from Self-Playcited by 64
  27. 2022Occupancy Flow Fields for Motion Forecasting in Autonomous Drivingcited by 62
  28. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  29. 2025A Survey of World Models for Autonomous Drivingcited by 60
  30. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  31. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  32. 2023Trajeglish: Traffic Modeling as Next-Token Predictioncited by 59
  33. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  34. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  35. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  36. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  37. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  38. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 51

50 most cited of 532 All 532 (JSON)