# Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications

arXiv: https://arxiv.org/abs/2311.17663 · HTML: https://mlsanity.com/paper/2311.17663/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 54 of its 59 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)
- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [Anisotropic Convolutional Networks for 3D Semantic Scene Completion](https://mlsanity.com/paper/2004.02122/) (2020)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)

## Built on by (most cited first, 30 of 52)

- [Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving](https://mlsanity.com/paper/2408.14197/) (2024)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model](https://mlsanity.com/paper/2410.10429/) (2024)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation](https://mlsanity.com/paper/2501.14729/) (2025)
- [GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding](https://mlsanity.com/paper/2412.13193/) (2024)
- [A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective](https://mlsanity.com/paper/2405.05173/) (2024)
- [RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar](https://mlsanity.com/paper/2405.14014/) (2024)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [Vision-based 3D occupancy prediction in autonomous driving: a review and outlook](https://mlsanity.com/paper/2405.02595/) (2024)
- [The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2502.10498/) (2025)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flow](https://mlsanity.com/paper/2502.17288/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions](https://mlsanity.com/paper/2411.07725/) (2024)
- [GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control](https://mlsanity.com/paper/2505.22421/) (2025)
- [MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Driving](https://mlsanity.com/paper/2503.15875/) (2025)
- [COME: Adding Scene-Centric Forecasting Control to Occupancy World Model](https://mlsanity.com/paper/2506.13260/) (2025)
- [Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving](https://mlsanity.com/paper/2403.08504/) (2024)
- [OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework](https://mlsanity.com/paper/2502.15180/) (2025)
- [Data Scaling Laws for Imitation Learning-Based End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.02689/) (2024)
- [VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction](https://mlsanity.com/paper/2506.05563/) (2025)
- [AGO: Adaptive Grounding for Open World 3D Occupancy Prediction](https://mlsanity.com/paper/2504.10117/) (2025)
- [ForecastOcc: Vision-based Semantic Occupancy Forecasting](https://mlsanity.com/paper/2602.08006/) (2026)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting](https://mlsanity.com/paper/2411.14169/) (2024)
- [ShelfOcc: Native 3D Supervision beyond LiDAR for Vision-Based Occupancy Estimation](https://mlsanity.com/paper/2511.15396/) (2025)
- [OccAny: Generalized Unconstrained Urban 3D Occupancy](https://mlsanity.com/paper/2603.23502/) (2026)
- [Delta-Triplane Transformers as Occupancy World Models](https://mlsanity.com/paper/2503.07338/) (2025)

## Abstract (arXiv)

Understanding how the surrounding environment changes is crucial for performing downstream tasks safely and reliably in autonomous driving applications. Recent occupancy estimation techniques using only camera images as input can provide dense occupancy representations of large-scale scenes based on the current observation. However, they are mostly limited to representing the current 3D space and do not consider the future state of surrounding objects along the time axis. To extend camera-only occupancy estimation into spatiotemporal prediction, we propose Cam4DOcc, a new benchmark for camera-only 4D occupancy forecasting, evaluating the surrounding scene changes in a near future. We build our benchmark based on multiple publicly available datasets, including nuScenes, nuScenes-Occupancy, and Lyft-Level5, which provides sequential occupancy states of general movable and static objects, as well as their 3D backward centripetal flow. To establish this benchmark for future research with comprehensive comparisons, we introduce four baseline types from diverse camera-based perception and prediction implementations, including a static-world occupancy model, voxelization of point cloud prediction, 2D-3D instance-based prediction, and our proposed novel end-to-end 4D occupancy forecasting network. Furthermore, the standardized evaluation protocol for preset multiple tasks is also provided to compare the performance of all the proposed baselines on present and future occupancy estimation with respect to objects of interest in autonomous driving scenarios. The dataset and our implementation of all four baselines in the proposed Cam4DOcc benchmark will be released here: https://github.com/haomo-ai/Cam4DOcc.