# Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction

arXiv: https://arxiv.org/abs/2407.07587 · HTML: https://mlsanity.com/paper/2407.07587/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 48 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 30 of 43)

- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation](https://mlsanity.com/paper/2307.01492/) (2023)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision](https://mlsanity.com/paper/2309.09502/) (2023)

## Built on by (most cited first, 24 of 24)

- [RenderWorld: World Model with Self-Supervised 3D Label](https://mlsanity.com/paper/2409.11356/) (2024)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction](https://mlsanity.com/paper/2504.19749/) (2025)
- [ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions](https://mlsanity.com/paper/2411.07725/) (2024)
- [An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training](https://mlsanity.com/paper/2412.13772/) (2024)
- [DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning](https://mlsanity.com/paper/2512.12799/) (2025)
- [GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2507.19451/) (2025)
- [AGO: Adaptive Grounding for Open World 3D Occupancy Prediction](https://mlsanity.com/paper/2504.10117/) (2025)
- [Semantic Causality-Aware Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2509.08388/) (2025)
- [OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Camera](https://mlsanity.com/paper/2511.03571/) (2025)
- [DIMO: Diverse 3D Motion Generation for Arbitrary Objects](https://mlsanity.com/paper/2511.07409/) (2025)
- [From 2D to 3D Cognition: A Brief Survey of General World Models](https://mlsanity.com/paper/2506.20134/) (2025)
- [MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies](https://mlsanity.com/paper/2501.15384/) (2025)
- [ODG: Occupancy Prediction Using Dual Gaussians](https://mlsanity.com/paper/2506.09417/) (2025)
- [Easy3D-Labels: Supervising Semantic Occupancy Estimation with 3D Pseudo-Labels for Automotive Perception](https://mlsanity.com/paper/2509.26087/) (2025)
- [InfraOcc: An Infrastructure Occupancy Benchmark with Static-to-Dynamic Reasoning](https://mlsanity.com/paper/2608.30657/) (2026)
- [BePo: Dual Representation for 3D Occupancy Prediction](https://mlsanity.com/paper/2506.07002/) (2025)
- [SelfOccFlow: Towards end-to-end self-supervised 3D Occupancy Flow prediction](https://mlsanity.com/paper/2602.23894/) (2026)
- [Class-Distribution Guided Active Learning for 3D Occupancy Prediction in Autonomous Driving](https://mlsanity.com/paper/2603.27294/) (2026)
- [GTAD: Global Temporal Aggregation Denoising Learning for 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2507.20963/) (2025)
- [LetOccVote: Learning Weakly Supervised 3D Occupancy through Consensus](https://mlsanity.com/paper/2609.04846/) (2026)
- [RoadOcc Learns When to Persist, Transport, or Refresh Memory for Roadside Occupancy Prediction](https://mlsanity.com/paper/2609.27677/) (2026)
- [TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception](https://mlsanity.com/paper/2609.32013/) (2026)
- [SelectOccFlow: Selective Spatiotemporal Aggregation for 3D Occupancy and Scene Flow Prediction](https://mlsanity.com/paper/2610.04356/) (2026)

## Abstract (arXiv)

Accurate perception of the dynamic environment is a fundamental task for autonomous driving and robot systems. This paper introduces Let Occ Flow, the first self-supervised work for joint 3D occupancy and occupancy flow prediction using only camera inputs, eliminating the need for 3D annotations. Utilizing TPV for unified scene representation and deformable attention layers for feature aggregation, our approach incorporates a novel attention-based temporal fusion module to capture dynamic object dependencies, followed by a 3D refine module for fine-gained volumetric representation. Besides, our method extends differentiable rendering to 3D volumetric flow fields, leveraging zero-shot 2D segmentation and optical flow cues for dynamic decomposition and motion optimization. Extensive experiments on nuScenes and KITTI datasets demonstrate the competitive performance of our approach over prior state-of-the-art methods. Our project page is available at https://eliliu2233.github.io/letoccflow/