mlsanityData to 7 Oct 2026

Visual Point Cloud Forecasting enables Scalable Autonomous Driving

arXiv 2312.17655 ↗First posted 29 Dec 2023cs.CVcited by 97

On the map: World models → World models for autonomous driving

Abstract (arXiv)

In contrast to extensive studies on general vision, pre-training for scalable visual autonomous driving remains seldom explored. Visual autonomous driving applications require features encompassing semantics, 3D geometry, and temporal information simultaneously for joint perception, prediction, and planning, posing dramatic challenges for pre-training. To resolve this, we bring up a new pre-training task termed as visual point cloud forecasting - predicting future point clouds from historical visual input. The key merit of this task captures the synergic learning of semantics, 3D structures, and temporal dynamics. Hence it shows superiority in various downstream tasks. To cope with this new problem, we present ViDAR, a general model to pre-train downstream visual encoders. It first extracts historical embeddings by the encoder. These representations are then transformed to 3D geometric space via a novel Latent Rendering operator for future point cloud prediction. Experiments show significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10% error reduction on motion forecasting, and ~15% less collision rate on planning.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 78 of its 98 references that are in the map. First posted on arXiv 29 Dec 2023.

Builds on

78 of 98 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  2. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  3. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  4. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  5. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2020Supervised Contrastive Learningcited by 2,803
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2017Deformable Convolutional Networkscited by 1,481
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  6. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  7. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  8. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  9. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  10. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  11. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  12. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  13. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  14. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  15. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  16. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  17. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  18. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  19. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  20. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  21. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  22. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  23. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  24. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  25. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 248
  26. 2023A Cookbook of Self-Supervised Learningcited by 248
  27. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  28. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  29. 2023Scene as Occupancycited by 208
  30. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  31. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  32. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  33. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  34. 2023SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videoscited by 149
  35. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  36. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  37. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  38. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 49; 2026: 22.

97 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  5. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  7. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  8. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  9. 20253D and 4D World Modeling: A Surveycited by 64
  10. 2025A Survey of World Models for Autonomous Drivingcited by 60
  11. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  12. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
38 more
  1. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  2. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  3. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  4. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  5. 2024Learning Manipulation by Predicting Interactioncited by 36
  6. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  8. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  9. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  10. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  11. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  12. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  13. 2025STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Predictioncited by 20
  14. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  15. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  16. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  17. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  18. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  19. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  20. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  21. 2024OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networkscited by 17
  22. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  23. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  24. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  25. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  26. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
  27. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 14
  28. 2025Self-Supervised Representation Learning with Joint Embedding Predictive Architecture for Automotive LiDAR Object Detectioncited by 13
  29. 2025ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Drivingcited by 12
  30. 2024GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Drivingcited by 11
  31. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  32. 2025SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queriescited by 10
  33. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  34. 2024World Models: The Safety Perspectivecited by 9
  35. 2025Self-Supervised Sparse Sensor Fusion for Long Range Perceptioncited by 8
  36. 20254D Visual Pre-training for Robot Learningcited by 8
  37. 2025Flux4D: Flow-based Unsupervised 4D Reconstructioncited by 7
  38. 2024Improving Distant 3D Object Detection Using 2D Box Supervisioncited by 6

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Driving
  2. 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
  3. 2026EponaV2: Driving World Model with Comprehensive Future Reasoning
  4. 2026ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model
  5. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
  6. 2026SUV: Future Scene Understanding as Video Generation for End-to-End Driving
  7. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting
  8. 2026ReDrive: Shaping Representations with World Modeling for End-to-End Driving