mlsanityData to 7 Oct 2026

Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting

arXiv 2302.13130 ↗First posted 25 Feb 2023cs.CV eess.SPcited by 76

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Predicting how the world can evolve in the future is crucial for motion planning in autonomous systems. Classical methods are limited because they rely on costly human annotations in the form of semantic class labels, bounding boxes, and tracks or HD maps of cities to plan their motion and thus are difficult to scale to large unlabeled datasets. One promising self-supervised task is 3D point cloud forecasting from unannotated LiDAR sequences. We show that this task requires algorithms to implicitly capture (1) sensor extrinsics (i.e., the egomotion of the autonomous vehicle), (2) sensor intrinsics (i.e., the sampling pattern specific to the particular LiDAR sensor), and (3) the shape and motion of other objects in the scene. But autonomous systems should make predictions about the world and not their sensors. To this end, we factor out (1) and (2) by recasting the task as one of spacetime (4D) occupancy forecasting. But because it is expensive to obtain ground-truth 4D occupancy, we render point cloud data from 4D occupancy predictions given sensor extrinsics and intrinsics, allowing one to train and test occupancy algorithms with unannotated LiDAR sequences. This also allows one to evaluate and compare point cloud forecasting algorithms across diverse datasets, sensors, and vehicles.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 13 of its 21 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

13 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct

Other references · 12

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2021Vision Transformers for Dense Predictioncited by 1,789
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  4. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  5. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  6. 2017Sparsity Invariant CNNscited by 325
  7. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  8. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  9. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  10. 2020Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Cloudscited by 116
  11. 2022Occupancy Flow Fields for Motion Forecasting in Autonomous Drivingcited by 62
  12. 2021Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networkscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 28; 2025: 28; 2026: 10.

76 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  5. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  6. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  7. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  8. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  9. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  10. 20253D and 4D World Modeling: A Surveycited by 64
  11. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  12. 2025A Survey of World Models for Autonomous Drivingcited by 60
38 more
  1. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
  2. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  3. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applicationscited by 50
  4. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 40
  5. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  6. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  7. 2024OccFusion: Multi-Sensor Fusion Framework for 3D Semantic Occupancy Predictioncited by 32
  8. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  9. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  10. 2023LiDAR-based 4D Occupancy Completion and Forecastingcited by 23
  11. 2024GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervisioncited by 21
  12. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  13. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  14. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  15. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  16. 2024Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologiescited by 17
  17. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  18. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 17
  19. 2024Neural Eulerian Scene Flow Fieldscited by 14
  20. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
  21. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 13
  22. 2025OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Frameworkcited by 11
  23. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  24. 2025SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queriescited by 10
  25. 2024SEED4D: A Synthetic Ego--Exo Dynamic 4D Data Generator, Driving Dataset and Benchmarkcited by 8
  26. 2024COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Drivingcited by 8
  27. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  28. 2025Self-Supervised Sparse Sensor Fusion for Long Range Perceptioncited by 8
  29. 2024Joint Perception and Prediction for Autonomous Driving: A Surveycited by 7
  30. 2025Flux4D: Flow-based Unsupervised 4D Reconstructioncited by 7
  31. 2023Semantic Complete Scene Forecasting from a 4D Dynamic Point Cloud Sequencecited by 6
  32. 2024CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understandingcited by 6
  33. 2024Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecastingcited by 6
  34. 2026ForecastOcc: Vision-based Semantic Occupancy Forecastingcited by 6
  35. 2024Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signalscited by 5
  36. 2025Delta-Triplane Transformers as Occupancy World Modelscited by 5
  37. 2026TruckDrive: Long-Range Autonomous Highway Driving Datasetcited by 5
  38. 2025GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Drivingcited by 4

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
  2. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
  3. 2026GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning
  4. 2024Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving
  5. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences
  6. 2026CloudDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain
  7. 2025OccStress: Stress-Testing the 4D Occupancy Forecasting Chain
  8. 2025B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding