# Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting

arXiv: https://arxiv.org/abs/2411.14169 · HTML: https://mlsanity.com/paper/2411.14169/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 38 of its 41 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting](https://mlsanity.com/paper/2302.13130/) (2023)
- [OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving](https://mlsanity.com/paper/2405.20337/) (2024)
- [Occupancy Flow Fields for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2203.03875/) (2022)
- [Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications](https://mlsanity.com/paper/2311.17663/) (2023)
- [StretchBEV: Stretching Future Instance Prediction Spatially and Temporally](https://mlsanity.com/paper/2203.13641/) (2022)
- [Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2110.04076/) (2021)
- [PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing](https://mlsanity.com/paper/1910.08287/) (2019)
- [UniOcc: Unifying Vision-Centric 3D Occupancy Prediction with Geometric and Semantic Rendering](https://mlsanity.com/paper/2306.09117/) (2023)

## Built on by (most cited first, 6 of 6)

- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [Delta-Triplane Transformers as Occupancy World Models](https://mlsanity.com/paper/2503.07338/) (2025)
- [SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model](https://mlsanity.com/paper/2511.22039/) (2025)
- [SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planning](https://mlsanity.com/paper/2601.06474/) (2026)
- [Dr.Occ: Depth- and Region-Guided 3D Occupancy from Surround-View Cameras for Autonomous Driving](https://mlsanity.com/paper/2603.01007/) (2026)
- [InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting](https://mlsanity.com/paper/2607.24431/) (2026)

## Abstract (arXiv)

The task of occupancy forecasting (OCF) involves utilizing past and present perception data to predict future occupancy states of autonomous vehicle surrounding environments, which is critical for downstream tasks such as obstacle avoidance and path planning. Existing 3D OCF approaches struggle to predict plausible spatial details for movable objects and suffer from slow inference speeds due to neglecting the bias and uneven distribution of changing occupancy states in both space and time. In this paper, we propose a novel spatiotemporal decoupling vision-based paradigm to explicitly tackle the bias and achieve both effective and efficient 3D OCF. To tackle spatial bias in empty areas, we introduce a novel spatial representation that decouples the conventional dense 3D format into 2D bird's-eye view (BEV) occupancy with corresponding height values, enabling 3D OCF derived only from 2D predictions thus enhancing efficiency. To reduce temporal bias on static voxels, we design temporal decoupling to improve end-to-end OCF by temporally associating instances via predicted flows. We develop an efficient multi-head network EfficientOCF to achieve 3D OCF with our devised spatiotemporally decoupled representation. A new metric, conditional IoU (C-IoU), is also introduced to provide a robust 3D OCF performance assessment, especially in datasets with missing or incomplete annotations. The experimental results demonstrate that EfficientOCF surpasses existing baseline methods on accuracy and efficiency, achieving state-of-the-art performance with a fast inference time of 82.33ms with a single GPU. Our code will be released as open source.