mlsanityData to 7 Oct 2026

Differentiable Raycasting for Self-supervised Occupancy Forecasting

arXiv 2210.01917 ↗First posted 4 Oct 2022cs.CV cs.ROcited by 83

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations such as freespace. Occupancy maps naturally disentangle the motion of the environment from the motion of the ego-vehicle. However, one cannot directly observe the full 3D occupancy of a scene (due to occlusion), making it difficult to use as a signal for learning. Our key insight is to use differentiable raycasting to "render" future occupancy predictions into future LiDAR sweep predictions, which can be compared with ground-truth sweeps for self-supervised learning. The use of differentiable raycasting allows occupancy to emerge as an internal representation within the forecasting network. In the absence of groundtruth occupancy, we quantitatively evaluate the forecasting of raycasted LiDAR sweeps and show improvements of upto 15 F1 points. For downstream motion planners, where emergent occupancy can be directly used to guide non-driveable regions, this representation relatively reduces the number of collisions with objects by up to 17% as compared to freespace-centric motion planners.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 20 of its 33 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

20 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 19

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  3. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  4. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  7. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  8. 2019Learning by Cheatingcited by 361
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  10. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  11. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  12. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
7 more
  1. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  2. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  3. 2019Just Go with the Flow: Self-Supervised Scene Flow Estimationcited by 98
  4. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  5. 2019What You See is What You Get: Exploiting Visibility for 3D Object Detectioncited by 69
  6. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  7. 20193D Multi-Object Tracking: A Baseline and New Evaluation Metricscited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 33; 2025: 27; 2026: 8.

83 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  4. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  5. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  6. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  7. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  8. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  9. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  10. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  11. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  12. 2023A Language Agent for Autonomous Drivingcited by 161
38 more
  1. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  2. 2024Behavior Generation with Latent Actionscited by 159
  3. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  4. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  5. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  6. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  7. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  8. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecastingcited by 76
  9. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  10. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  11. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  12. 20253D and 4D World Modeling: A Surveycited by 64
  13. 2024Embodied Understanding of Driving Scenarioscited by 63
  14. 2025A Survey of World Models for Autonomous Drivingcited by 60
  15. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  16. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applicationscited by 50
  17. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  18. 2024Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Drivingcited by 46
  19. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
  20. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  21. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 40
  22. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  23. 2023PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Drivingcited by 33
  24. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 27
  25. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  26. 2023LiDAR-based 4D Occupancy Completion and Forecastingcited by 23
  27. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  28. 2024GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervisioncited by 21
  29. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  30. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  31. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  32. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  33. 2024DeepInteraction++: Multi-Modality Interaction for Autonomous Drivingcited by 15
  34. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  35. 2025UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertaintycited by 14
  36. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 13
  37. 2025DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Modelcited by 13
  38. 2024Autonomous Driving with Spiking Neural Networkscited by 12

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
  2. 2026GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning
  3. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
  4. 2024Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving
  5. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction
  6. 2024Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting
  7. 2025OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework
  8. 2025OccStress: Stress-Testing the 4D Occupancy Forecasting Chain