mlsanityData to 7 Oct 2026

RenderWorld: World Model with Self-Supervised 3D Label

arXiv 2409.11356 ↗First posted 17 Sep 2024cs.CV cs.AIcited by 44

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

48 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  2. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  3. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving27 Apr
  4. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct
  5. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 43

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2018World Modelscited by 1,444
  6. 2022Planning-oriented Autonomous Drivingcited by 1,068
  7. 2016End to End Learning for Self-Driving Carscited by 1,000
  8. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  9. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  10. 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 684
  11. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  12. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
31 more
  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  3. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
  4. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  5. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  6. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  7. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  8. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  9. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  10. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  11. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  12. 2023Scene as Occupancycited by 208
  13. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  14. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 162
  15. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  16. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  17. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  18. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  19. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 126
  20. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  21. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 114
  22. 2018TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agentscited by 111
  23. 2024GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splattingcited by 50
  24. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  25. 2022CL3D: Unsupervised Domain Adaptation for Cross-LiDAR 3D Detectioncited by 26
  26. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24
  27. 2024Let Occ Flow: Self-Supervised 3D Occupancy Flow Predictioncited by 23
  28. 2024OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flowcited by 22
  29. 2024HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Predictioncited by 12
  30. 2023Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learningcited by 7
  31. 2023One Training for Multiple Deployments: Polar-based Adaptive BEV Perception for Autonomous Drivingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 31; 2026: 9.

44 papers.

Sort:

By area:

  1. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  2. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  3. 20253D and 4D World Modeling: A Surveycited by 64
  4. 2025A Survey of World Models for Autonomous Drivingcited by 60
  5. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  6. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  7. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  8. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  9. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  10. 2026Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentationcited by 23
  11. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 20
  12. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
32 more
  1. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 19
  2. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  3. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  4. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 14
  5. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
  6. 2025Four Principles for Physically Interpretable World Modelscited by 10
  7. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  8. 2025AGO: Adaptive Grounding for Open World 3D Occupancy Predictioncited by 9
  9. 2025Occupancy World Model for Robotscited by 8
  10. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  11. 2025Semantic Causality-Aware Vision-Based 3D Occupancy Predictioncited by 8
  12. 2025ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptationcited by 7
  13. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  14. 2025Delta-Triplane Transformers as Occupancy World Modelscited by 5
  15. 2025WPT: World-to-Policy Transfer via Online World Model Distillationcited by 4
  16. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  17. 2024Physically Interpretable World Models via Weakly Supervised Representation Learningcited by 3
  18. 2025GM-MoE: Low-Light Enhancement with Gated-Mechanism Mixture-of-Expertscited by 3
  19. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  20. 2025Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Modelscited by 3
  21. 2025SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Modelcited by 3
  22. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  23. 2026A Definition and Roadmap for World Modelscited by 3
  24. 2025Easy3D-Labels: Supervising Semantic Occupancy Estimation with 3D Pseudo-Labels for Automotive Perceptioncited by 2
  25. 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Drivingcited by 2
  26. 2025GP-GS: Gaussian Processes Densification for 3D Gaussian Splattingcited by 1
  27. 2025HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Drivingcited by 1
  28. 2026Unpaired Image Deraining Using Reward-Guided Self-Reinforcement Strategycited by 1
  29. 2026GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planningcited by 0
  30. 2026CascadeOcc: Rethinking 3D Occupancy World Models with Cascaded VQ Representationscited by 0
  31. 2026Geometry-Aware Spatio-Temporal Context Modeling for 4D Occupancy Forecastingcited by 0
  32. 2026Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
  2. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
  3. 2023UniWorld: Autonomous Driving Pre-training via World Models
  4. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting
  5. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Driving
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
  7. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
  8. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving