mlsanityData to 7 Oct 2026

HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

arXiv 2501.14729 ↗First posted 24 Jan 2025cs.CVcited by 56

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model, enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 69 of its 81 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

69 of 81 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Model12 Dec
  2. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model14 Oct
  3. 2024RenderWorld: World Model with Self-Supervised 3D Label17 Sep
  4. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving5 Sep
  5. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents8 Jul
  6. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting12 Jun
  7. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  9. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  10. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  11. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  12. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
9 more
  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  2. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications29 Nov
  3. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  4. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  5. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  6. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  7. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  8. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
36 more
  1. 2018World Modelscited by 1,444
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2022Planning-oriented Autonomous Drivingcited by 1,068
  4. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  7. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  9. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  10. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  12. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  13. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  14. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  15. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  16. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  17. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  18. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  19. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  20. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  21. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  22. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  23. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  24. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  25. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  26. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  27. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  28. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  29. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  30. 2024Embodied Understanding of Driving Scenarioscited by 63
  31. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  32. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  33. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  34. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  35. 2023DDS3D: Dense Pseudo-Labels with Dynamic Threshold for Semi-Supervised 3D Object Detectioncited by 18
  36. 2024Make Your ViT-based Multi-view 3D Detectors Faster via Token Compressioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 32.

56 papers.

Sort:

By area:

  1. 2025Motus: A Unified Latent Action World Modelcited by 231
  2. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  4. 20253D and 4D World Modeling: A Surveycited by 64
  5. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  6. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  7. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  8. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  9. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  10. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  11. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  12. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
38 more
  1. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  2. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  3. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20
  4. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  5. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  6. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  7. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  8. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 17
  9. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  10. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  11. 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoningcited by 13
  12. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  13. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  14. 2025GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generationcited by 10
  15. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  16. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  17. 2026TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMcited by 9
  18. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  19. 2025Self-Supervised Sparse Sensor Fusion for Long Range Perceptioncited by 8
  20. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  21. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  22. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Drivingcited by 5
  23. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  24. 2026SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planningcited by 4
  25. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Modelscited by 4
  26. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
  27. 2026SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Drivingcited by 4
  28. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  29. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 3
  30. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  31. 2026A Definition and Roadmap for World Modelscited by 3
  32. 2026PROSPECT: Unified Streaming Vision-Language Navigation via Semantic--Spatial Fusion and Latent Predictive Representationcited by 2
  33. 2026PointTPA: Dynamic Network Parameter Adaptation for 3D Scene Understandingcited by 2
  34. 2026ECG-WM: A Physiology-Informed ECG World Model for Clinical Intervention Simulationcited by 2
  35. 2026DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Drivingcited by 1
  36. 2026FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Modelcited by 1
  37. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representationscited by 0
  38. 2026WorldVLM: Combining World Model Forecasting and Vision-Language Reasoningcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
  2. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
  3. 2026UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning
  4. 2026Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
  5. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving
  6. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
  7. 2026EponaV2: Driving World Model with Comprehensive Future Reasoning
  8. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Driving