mlsanityData to 7 Oct 2026

Enhancing End-to-End Autonomous Driving with Latent World Model

arXiv 2406.08481 ↗First posted 12 Jun 2024cs.CVcited by 161

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

In autonomous driving, end-to-end planners directly utilize raw sensor data, enabling them to extract richer scene features and reduce information loss compared to traditional planners. This raises a crucial research question: how can we develop better scene feature representations to fully leverage sensor data in end-to-end driving? Self-supervised learning methods show great success in learning rich feature representations in NLP and computer vision. Inspired by this, we propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving. LAW predicts future scene features based on current features and ego trajectories. This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks, improving scene feature learning and optimizing trajectory prediction. LAW achieves state-of-the-art performance across multiple benchmarks, including real-world open-loop benchmark nuScenes, NAVSIM, and simulator-based closed-loop benchmark CARLA. The code is released at https://github.com/BraveGroup/LAW.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

52 of 63 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  2. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  3. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  4. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  5. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  6. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  7. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct
  8. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar
  9. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  10. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 42

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  9. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  10. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  11. 2020Tracking Objects as Pointscited by 483
  12. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
30 more
  1. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  2. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  3. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  4. 2019Learning by Cheatingcited by 361
  5. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  6. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  7. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  8. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  9. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  10. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  11. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  12. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  13. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  14. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  15. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  16. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  17. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  18. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  19. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  20. 2022Learning from All Vehiclescited by 135
  21. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  22. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 114
  23. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  24. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  25. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  26. 2021Immortal Tracker: Tracklet Never Diescited by 42
  27. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  28. 2023Fully Sparse Fusion for 3D Object Detectioncited by 29
  29. 2022Densely Constrained Depth Estimator for Monocular 3D Object Detectioncited by 27
  30. 2023Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modelingcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 62; 2026: 92.

161 papers.

Sort:

By area:

  1. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  3. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  4. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  5. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  6. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  7. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  8. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  9. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 76
  10. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  11. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  12. 2025A Survey of World Models for Autonomous Drivingcited by 60
38 more
  1. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  2. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  3. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  4. 2024Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Drivingcited by 46
  5. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  6. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  7. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  8. 2025iPad: Iterative Proposal-centric End-to-End Autonomous Drivingcited by 41
  9. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  10. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  13. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  14. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  15. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  16. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 26
  17. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  18. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  19. 2025StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Drivingcited by 20
  20. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  21. 2025WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Drivingcited by 19
  22. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  23. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  24. 2025DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Drivingcited by 17
  25. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  26. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  27. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  28. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  29. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  30. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Drivingcited by 14
  31. 2026ResWorld: Temporal Residual World Model for End-to-End Autonomous Drivingcited by 14
  32. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 13
  33. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  34. 2026AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuningcited by 13
  35. 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memorycited by 13
  36. 2025EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Drivingcited by 12
  37. 2025MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Drivingcited by 12
  38. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 12

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Review
  2. 2023End-to-end Autonomous Driving: Challenges and Frontiers
  3. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model
  4. 2026PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning
  5. 2022Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving
  6. 2025LAP: Fast LAtent Diffusion Planner for Autonomous Driving
  7. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey
  8. 2022SelfD: Self-Learning Large-Scale Driving Policies From the Web