mlsanityData to 7 Oct 2026

One Thousand and One Hours: Self-driving Motion Prediction Dataset

arXiv 2006.14480 ↗First posted 25 Jun 2020cs.CV cs.LG cs.ROcited by 239

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Motivated by the impact of large-scale datasets on ML systems we present the largest self-driving dataset for motion prediction to date, containing over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California, over a four-month period. It consists of 170,000 scenes, where each scene is 25 seconds long and captures the perception output of the self-driving system, which encodes the precise positions and motions of nearby vehicles, cyclists, and pedestrians over time. On top of this, the dataset contains a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area. We show that using a dataset of this size dramatically improves performance for key self-driving problems. Combined with the provided software kit, this collection forms the largest and most detailed dataset to date for the development of self-driving machine learning tasks, such as motion forecasting, motion planning and simulation. The full dataset is available at http://level5.lyft.com/.

Key facts. Cited by 239 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 1 of its 27 references that are in the map. First posted on arXiv 25 Jun 2020.

Builds on

1 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks18 Sep

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 40; 2023: 50; 2024: 48; 2025: 46; 2026: 26.

239 papers.

Sort:

By area:

  1. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  2. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  3. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  4. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  5. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  6. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  7. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 165
  8. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  9. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  10. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  11. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  12. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
38 more
  1. 2023V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecastingcited by 114
  2. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  3. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  4. 2021Explanations in Autonomous Driving: A Surveycited by 80
  5. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  6. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  7. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  8. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  9. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  10. 20253D and 4D World Modeling: A Surveycited by 64
  11. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  12. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  13. 2022Uncertainty-aware Vision-based Metric Cross-view Geolocalizationcited by 54
  14. 2024P-MapNet: Far-seeing Map Generator Enhanced by both SDMap and HDMap Priorscited by 54
  15. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  16. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 51
  17. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  18. 2021Scalable Scene Flow from Point Clouds in the Real Worldcited by 49
  19. 2021ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenationcited by 41
  20. 2022A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Researchcited by 40
  21. 2025Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Cameracited by 38
  22. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  23. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  24. 2022How many Observations are Enough? Knowledge Distillation for Trajectory Forecastingcited by 35
  25. 2021Pseudo-labeling for Scalable 3D Object Detectioncited by 32
  26. 2021Imagining The Road Ahead: Multi-Agent Trajectory Prediction via Differentiable Simulationcited by 32
  27. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  28. 2024Language-Image Models with 3D Understandingcited by 31
  29. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
  30. 2021Autonomy 2.0: Why is self-driving always 5 years away?cited by 30
  31. 2022SIND: A Drone Dataset at Signalized Intersection in Chinacited by 29
  32. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  33. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  34. 2022Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Expertscited by 28
  35. 2023WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecastingcited by 28
  36. 2023Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Datasetcited by 28
  37. 2022Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Modelscited by 27
  38. 2022Trust, but Verify: Cross-Modality Fusion for HD Map Change Detectioncited by 25

50 most cited of 239 All 239 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving
  2. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving
  3. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
  4. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving
  5. 2026How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models
  6. 2024Data Scaling Laws for Imitation Learning-Based End-to-End Autonomous Driving
  7. 2019Predicting Motion of Vulnerable Road Users using High-Definition Maps and Efficient ConvNets
  8. 2024TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks