mlsanityData to 14 Oct 2026

MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations

arXiv 2311.11762 ↗First posted 20 Nov 2023cs.LG cs.ROcited by 33

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World models for autonomous driving have the potential to dramatically improve the reasoning capabilities of today's systems. However, most works focus on camera data, with only a few that leverage lidar data or combine both to better represent autonomous vehicle sensor setups. In addition, raw sensor predictions are less actionable than 3D occupancy predictions, but there are no works examining the effects of combining both multimodal sensor data and 3D occupancy prediction. In this work, we perform a set of experiments with a MUltimodal World Model with Geometric VOxel representations (MUVO) to evaluate different sensor fusion strategies to better understand the effects on sensor data prediction. We also analyze potential weaknesses of current sensor fusion approaches and examine the benefits of additionally predicting 3D occupancy.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 117 of its 153 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

117 of 153 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  2. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  3. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  4. 2023UniWorld: Autonomous Driving Pre-training via World Models14 Aug
  5. 2023Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving10 Aug
  6. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving27 Apr
  7. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction7 Mar
  8. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  9. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct
  10. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  11. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 106

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
38 more
  1. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  11. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  12. 2022A Generalist Agentcited by 808
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  14. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  15. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  16. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  17. 2019Model-Based Reinforcement Learning for Ataricited by 502
  18. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  19. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  20. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  21. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  22. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 410
  23. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  24. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  25. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  26. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  27. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  28. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  29. 2022Transformers are Sample-Efficient World Modelscited by 328
  30. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  31. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  32. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  33. 2021Voxel Transformer for 3D Object Detectioncited by 247
  34. 2021End-to-end Interpretable Neural Motion Plannercited by 242
  35. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  36. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  37. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  38. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 10; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  3. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  4. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  5. 2024World Models for Autonomous Driving: An Initial Surveycited by 80
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  8. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  9. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  10. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  11. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  12. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
21 more
  1. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  2. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 21
  3. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  4. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 18
  5. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 16
  6. 2024AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Drivingcited by 14
  7. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  8. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  9. 2024World Models: The Safety Perspectivecited by 9
  10. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  11. 20246Img-to-3D: Few-Image Large-Scale Outdoor Driving Scene Reconstructioncited by 5
  12. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Modelscited by 4
  13. 2024UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Drivingcited by 3
  14. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  15. 2024How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perceptioncited by 2
  16. 2026AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Modelscited by 2
  17. 2025Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMscited by 1
  18. 2024Data-driven Camera and Lidar Simulation Models for Autonomous Driving: A Review from Generative Models to Volume Rendererscited by 0
  19. 2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0
  20. 2026Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Trafficcited by 0
  21. 2026PhysWAM: Physically Consistent World Action Model for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
  2. 2025Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving
  3. 2026Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
  4. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Driving
  5. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
  6. 2026The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset
  7. 2024Probing Multimodal LLMs as World Models for Driving
  8. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles