# WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecasting

arXiv: https://arxiv.org/abs/2304.03834 · HTML: https://mlsanity.com/paper/2304.03834/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2210.07372/) (2022)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [What-If Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2008.10587/) (2020)

## Built on by (most cited first, 28 of 28)

- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [Solving Motion Planning Tasks with a Scalable Generative Model](https://mlsanity.com/paper/2407.02797/) (2024)
- [Language-Driven Interactive Traffic Trajectory Generation](https://mlsanity.com/paper/2405.15388/) (2024)
- [WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving](https://mlsanity.com/paper/2407.04281/) (2024)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model](https://mlsanity.com/paper/2409.09575/) (2024)
- [LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation](https://mlsanity.com/paper/2504.11521/) (2025)
- [LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends](https://mlsanity.com/paper/2410.15281/) (2024)
- [MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR](https://mlsanity.com/paper/2312.02409/) (2023)
- [MoST: Multi-modality Scene Tokenization for Motion Prediction](https://mlsanity.com/paper/2404.19531/) (2024)
- [Highly Accurate and Diverse Traffic Data: The DeepScenario Open 3D Dataset](https://mlsanity.com/paper/2504.17371/) (2025)
- [HeLiMOS: A Dataset for Moving Object Segmentation in 3D Point Clouds From Heterogeneous LiDAR Sensors](https://mlsanity.com/paper/2408.06328/) (2024)
- [SwapTransformer: highway overtaking tactical planner model via imitation learning on OSHA dataset](https://mlsanity.com/paper/2401.01425/) (2024)
- [DECODE: Domain-aware Continual Domain Expansion for Motion Prediction](https://mlsanity.com/paper/2411.17917/) (2024)
- [Can the Waymo Open Motion Dataset Support Realistic Behavioral Modeling? A Validation Study with Naturalistic Trajectories](https://mlsanity.com/paper/2509.03515/) (2025)
- [nuScenes Revisited: Progress and Challenges in Autonomous Driving](https://mlsanity.com/paper/2512.02448/) (2025)
- [AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models](https://mlsanity.com/paper/2603.28963/) (2026)
- [Evidence and quantification of cooperation of driving agents in mixed traffic flow](https://mlsanity.com/paper/2408.07297/) (2024)
- [HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic](https://mlsanity.com/paper/2602.03447/) (2026)
- [Learning Driver Models for Automated Vehicles via Knowledge Sharing and Personalization](https://mlsanity.com/paper/2308.16870/) (2023)
- [LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integration](https://mlsanity.com/paper/2410.15819/) (2024)
- [Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehicles](https://mlsanity.com/paper/2504.01336/) (2025)
- [Beyond Features: How Dataset Design Influences Multi-Agent Trajectory Prediction Performance](https://mlsanity.com/paper/2507.05098/) (2025)
- [An Empirical Markov Chain Car-Following (MC-CF) Model](https://mlsanity.com/paper/2603.27909/) (2026)
- [Conditional Flow-VAE for Safety-Critical Traffic Scenario Generation](https://mlsanity.com/paper/2605.04366/) (2026)
- [Vision-Based Runtime Monitoring under Varying Specifications using Semantic Latent Representations](https://mlsanity.com/paper/2605.13923/) (2026)
- [Driving, Fast or Slow? Neuro-Symbolic Guidance for Motion Prediction in Multi-Modal Ground Mobility](https://mlsanity.com/paper/2606.15251/) (2026)
- [NSF-HRPT: Neural Semantic Field meets Hierarchical Risk Perception Tree for Safety-Critical Scenario Assessment](https://mlsanity.com/paper/2608.04776/) (2026)

## Abstract (arXiv)

Widely adopted motion forecasting datasets substitute the observed sensory inputs with higher-level abstractions such as 3D boxes and polylines. These sparse shapes are inferred through annotating the original scenes with perception systems' predictions. Such intermediate representations tie the quality of the motion forecasting models to the performance of computer vision models. Moreover, the human-designed explicit interfaces between perception and motion forecasting typically pass only a subset of the semantic information present in the original sensory input. To study the effect of these modular approaches, design new paradigms that mitigate these limitations, and accelerate the development of end-to-end motion forecasting models, we augment the Waymo Open Motion Dataset (WOMD) with large-scale, high-quality, diverse LiDAR data for the motion forecasting task. The new augmented dataset WOMD-LiDAR consists of over 100,000 scenes that each spans 20 seconds, consisting of well-synchronized and calibrated high quality LiDAR point clouds captured across a range of urban and suburban geographies (https://waymo.com/open/data/motion/). Compared to Waymo Open Dataset (WOD), WOMD-LiDAR dataset contains 100x more scenes. Furthermore, we integrate the LiDAR data into the motion forecasting model training and provide a strong baseline. Experiments show that the LiDAR data brings improvement in the motion forecasting task. We hope that WOMD-LiDAR will provide new opportunities for boosting end-to-end motion forecasting models.