# Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving

arXiv: https://arxiv.org/abs/2308.01471 · HTML: https://mlsanity.com/paper/2308.01471/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Placed in

- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)

## Built on by (most cited first, 30 of 30)

- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [UnO: Unsupervised Occupancy Fields for Perception and Forecasting](https://mlsanity.com/paper/2406.08691/) (2024)
- [RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar](https://mlsanity.com/paper/2405.14014/) (2024)
- [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision](https://mlsanity.com/paper/2405.10591/) (2024)
- [DeTra: A Unified Model for Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2406.04426/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2404.01486/) (2024)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation](https://mlsanity.com/paper/2311.01446/) (2023)
- [SurroundSDF: Implicit 3D Scene Understanding Based on Signed Distance Field](https://mlsanity.com/paper/2403.14366/) (2024)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)
- [RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments](https://mlsanity.com/paper/2408.15503/) (2024)
- [LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds](https://mlsanity.com/paper/2311.01444/) (2023)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding](https://mlsanity.com/paper/2406.14422/) (2024)
- [SceneMotion: From Agent-Centric Embeddings to Scene-Wide Forecasts](https://mlsanity.com/paper/2408.01537/) (2024)
- [GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Driving](https://mlsanity.com/paper/2503.15672/) (2025)
- [Vestibular schwannoma growth prediction from longitudinal MRI by time conditioned neural fields](https://mlsanity.com/paper/2404.02614/) (2024)
- [Instantaneous Perception of Moving Objects in 3D](https://mlsanity.com/paper/2405.02781/) (2024)
- [QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy](https://mlsanity.com/paper/2511.17221/) (2025)
- [Predicting Long-horizon Futures by Conditioning on Geometry and Time](https://mlsanity.com/paper/2404.11554/) (2024)
- [VideoGAN-based Trajectory Proposal for Automated Vehicles](https://mlsanity.com/paper/2506.16209/) (2025)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [Vehicle Dynamics Embedded World Models for Autonomous Driving](https://mlsanity.com/paper/2512.02417/) (2025)
- [Visual Implicit Geometry Transformer for Autonomous Driving](https://mlsanity.com/paper/2602.05573/) (2026)
- [MC-DeTra: Motion-Consistent Joint Object Detection and Socially-Aware Trajectory Forecasting in Bird's-Eye-View Images](https://mlsanity.com/paper/2609.11717/) (2026)
- [TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception](https://mlsanity.com/paper/2609.32013/) (2026)

## Abstract (arXiv)

A self-driving vehicle (SDV) must be able to perceive its surroundings and predict the future behavior of other traffic participants. Existing works either perform object detection followed by trajectory forecasting of the detected objects, or predict dense occupancy and flow grids for the whole scene. The former poses a safety concern as the number of detections needs to be kept low for efficiency reasons, sacrificing object recall. The latter is computationally expensive due to the high-dimensionality of the output grid, and suffers from the limited receptive field inherent to fully convolutional networks. Furthermore, both approaches employ many computational resources predicting areas or objects that might never be queried by the motion planner. This motivates our unified approach to perception and future prediction that implicitly represents occupancy and flow over time with a single neural network. Our method avoids unnecessary computation, as it can be directly queried by the motion planner at continuous spatio-temporal locations. Moreover, we design an architecture that overcomes the limited receptive field of previous explicit occupancy prediction methods by adding an efficient yet effective global attention mechanism. Through extensive experiments in both urban and highway settings, we demonstrate that our implicit model outperforms the current state-of-the-art. For more information, visit the project website: https://waabi.ai/research/implicito.