mlsanityData to 14 Oct 2026

WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios

arXiv 2510.26125 ↗First posted 30 Oct 2025cs.CV cs.AIcited by 62

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Vision-based end-to-end (E2E) driving has garnered significant interest in the research community due to its scalability and synergy with multimodal large language models (MLLMs). However, current E2E driving benchmarks primarily feature nominal scenarios, failing to adequately test the true potential of these systems. Furthermore, existing open-loop evaluation metrics often fall short in capturing the multi-modal nature of driving or effectively evaluating performance in long-tail scenarios. To address these gaps, we introduce the Waymo Open Dataset for End-to-End Driving (WOD-E2E). WOD-E2E contains 4,021 driving segments (approximately 12 hours), specifically curated for challenging long-tail scenarios that that are rare in daily life with an occurring frequency of less than 0.03%. Concretely, each segment in WOD-E2E includes the high-level routing information, ego states, and 360-degree camera views from 8 surrounding cameras. To evaluate the E2E driving performance on these long-tail situations, we propose a novel open-loop evaluation metric: Rater Feedback Score (RFS). Unlike conventional metrics that measure the distance between predicted way points and the logs, RFS measures how closely the predicted trajectory matches rater-annotated trajectory preference labels. We have released rater preference labels for all WOD-E2E validation set segments, while the held out test set labels have been used for the 2025 WOD-E2E Challenge. Through our work, we aim to foster state of the art research into generalizable, robust, and safe end-to-end autonomous driving agents capable of handling complex real-world situations.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 40 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

34 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving18 Aug
  2. 2025HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios6 Jun
  3. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation30 May
  4. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving19 Dec
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 28

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  7. 2022Planning-oriented Autonomous Drivingcited by 1,087
  8. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  9. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  10. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  11. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  12. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
16 more
  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  2. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  4. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  5. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  6. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 212
  7. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  8. 2023V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perceptioncited by 155
  9. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  10. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  11. 2022CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detectioncited by 30
  12. 2025AdaWM: Adaptive World Model based Planning for Autonomous Drivingcited by 24
  13. 2024CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flowcited by 14
  14. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  15. 2024MoST: Multi-modality Scene Tokenization for Motion Predictioncited by 8
  16. 2025Using Ensemble Diffusion to Estimate Uncertainty for End-to-End Autonomous Drivingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 49.

62 papers.

Sort:

By area:

  1. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 36
  2. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  3. 2025LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Drivingcited by 23
  4. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 17
  5. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  6. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  7. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  8. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 9
  9. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Drivingcited by 9
  10. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 9
  11. 2026MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Drivingcited by 6
  12. 2025NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generationcited by 5
38 more
  1. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Drivingcited by 5
  2. 2026TruckDrive: Long-Range Autonomous Highway Driving Datasetcited by 5
  3. 2026Pre-VLA: Preemptive Runtime Verification for Reliable Vision-Language-Action and World-Model Rolloutscited by 5
  4. 2026Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Drivingcited by 5
  5. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  6. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  7. 2025LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agentscited by 3
  8. 2026ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Drivingcited by 3
  9. 2026123D: Unifying Multi-Modal Autonomous Driving Data at Scalecited by 3
  10. 2026DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Modelscited by 3
  11. 2026NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Drivingcited by 2
  12. 2026Post-Training in End-to-End Autonomous Drivingcited by 2
  13. 2026OpenLongTail: Generative Scaling of Long-Tail Driving Datacited by 2
  14. 2026SUV: Future Scene Understanding as Video Generation for End-to-End Drivingcited by 2
  15. 2025Closing the Navigation Compliance Gap in End-to-end Autonomous Drivingcited by 1
  16. 2025Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensuscited by 1
  17. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  18. 2026Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Drivingcited by 1
  19. 2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
  20. 2026Action Emergence from Streaming Intentcited by 1
  21. 2026Driving Intents Amplify Planning-Oriented Reinforcement Learningcited by 1
  22. 2026A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Modelscited by 1
  23. 2026Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoningcited by 1
  24. 2025SAVeD: A First-Person Social Media Video Dataset for ADAS-equipped vehicle Near-Miss and Crash Event Analysescited by 0
  25. 2025Are All Data Necessary? Efficient Data Pruning for Large-scale Autonomous Driving Dataset via Trajectory Entropy Maximizationcited by 0
  26. 2026Kinematics-Aware Latent World Models for Data-Efficient Autonomous Drivingcited by 0
  27. 2026Fusing Driver Perceived and Physical Risk for Safety Critical Scenario Screening in Autonomous Drivingcited by 0
  28. 2026ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Predictioncited by 0
  29. 2026SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Drivingcited by 0
  30. 2026OVPD: A Virtual-Physical Fusion Testing Dataset of OnSite Auton-omous Driving Challengecited by 0
  31. 2026Dynamics Distillation for Efficient and Transferable Control Learningcited by 0
  32. 2026Temporal Sampling Frequency Matters: A Capacity-Aware Study of End-to-End Driving Trajectory Predictioncited by 0
  33. 2026DriveSafer: End-to-End Autonomous Driving with Safety Guidancecited by 0
  34. 2026X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understandingcited by 0
  35. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Drivingcited by 0
  36. 2026StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasetscited by 0
  37. 2026Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videoscited by 0
  38. 2026Is Your Trajectory Displacement Safe in Long-tail?cited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Hidden Biases of End-to-End Driving Models
  2. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
  3. 2026Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes
  4. 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving
  5. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
  6. 2024Probing Multimodal LLMs as World Models for Driving
  7. 2026What Matters for Scalable and Robust Learning in End-to-End Driving Planners?
  8. 2024Hidden Biases of End-to-End Driving Datasets