# Occupancy Flow Fields for Motion Forecasting in Autonomous Driving

arXiv: https://arxiv.org/abs/2203.03875 · HTML: https://mlsanity.com/paper/2203.03875/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [HDNET: Exploiting HD Maps for 3D Object Detection](https://mlsanity.com/paper/2012.11704/) (2020)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [CAR-Net: Clairvoyant Attentive Recurrent Network](https://mlsanity.com/paper/1711.10061/) (2017)
- [Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction](https://mlsanity.com/paper/1910.08041/) (2019)
- [PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving](https://mlsanity.com/paper/2003.08744/) (2020)

## Built on by (most cited first, 30 of 62)

- [Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting](https://mlsanity.com/paper/2302.13130/) (2023)
- [A Survey of World Models for Autonomous Driving](https://mlsanity.com/paper/2501.11260/) (2025)
- [Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders](https://mlsanity.com/paper/2206.09900/) (2022)
- [Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications](https://mlsanity.com/paper/2311.17663/) (2023)
- [V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models](https://mlsanity.com/paper/2408.09251/) (2024)
- [UnO: Unsupervised Occupancy Fields for Perception and Forecasting](https://mlsanity.com/paper/2406.08691/) (2024)
- [Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving](https://mlsanity.com/paper/2308.01471/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [PowerBEV: A Powerful Yet Lightweight Framework for Instance Prediction in Bird's-Eye View](https://mlsanity.com/paper/2306.10761/) (2023)
- [LiDAR-based 4D Occupancy Completion and Forecasting](https://mlsanity.com/paper/2310.11239/) (2023)
- [PointBeV: A Sparse Approach to BeV Predictions](https://mlsanity.com/paper/2312.00703/) (2023)
- [GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision](https://mlsanity.com/paper/2405.10591/) (2024)
- [Hybrid-Prediction Integrated Planning for Autonomous Driving](https://mlsanity.com/paper/2402.02426/) (2024)
- [Language-Driven Active Learning for Diverse Open-Set 3D Object Detection](https://mlsanity.com/paper/2404.12856/) (2024)
- [DeTra: A Unified Model for Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2406.04426/) (2024)
- [Uncertainty estimation for Cross-dataset performance in Trajectory prediction](https://mlsanity.com/paper/2205.07310/) (2022)
- [HOPE: Hierarchical Spatial-temporal Network for Occupancy Flow Prediction](https://mlsanity.com/paper/2206.10118/) (2022)
- [QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2404.01486/) (2024)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation](https://mlsanity.com/paper/2311.01446/) (2023)
- [UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarios](https://mlsanity.com/paper/2405.03807/) (2024)
- [STrajNet: Multi-modal Hierarchical Transformer for Occupancy Flow Field Prediction in Autonomous Driving](https://mlsanity.com/paper/2208.00394/) (2022)
- [Occupancy Prediction-Guided Neural Planner for Autonomous Driving](https://mlsanity.com/paper/2305.03303/) (2023)
- [OFMPNet: Deep End-to-End Model for Occupancy and Flow Prediction in Urban Environment](https://mlsanity.com/paper/2404.02263/) (2024)
- [Bridging the Gap Between Multi-Step and One-Shot Trajectory Prediction via Self-Supervision](https://mlsanity.com/paper/2306.03367/) (2023)
- [LabelFormer: Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds](https://mlsanity.com/paper/2311.01444/) (2023)
- [Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving](https://mlsanity.com/paper/2407.21126/) (2024)
- [VectorFlow: Combining Images and Vectors for Traffic Occupancy and Flow Prediction](https://mlsanity.com/paper/2208.04530/) (2022)
- [FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding](https://mlsanity.com/paper/2406.14422/) (2024)

## Abstract (arXiv)

We propose Occupancy Flow Fields, a new representation for motion forecasting of multiple agents, an important task in autonomous driving. Our representation is a spatio-temporal grid with each grid cell containing both the probability of the cell being occupied by any agent, and a two-dimensional flow vector representing the direction and magnitude of the motion in that cell. Our method successfully mitigates shortcomings of the two most commonly-used representations for motion forecasting: trajectory sets and occupancy grids. Although occupancy grids efficiently represent the probabilistic location of many agents jointly, they do not capture agent motion and lose the agent identities. To this end, we propose a deep learning architecture that generates Occupancy Flow Fields with the help of a new flow trace loss that establishes consistency between the occupancy and flow predictions. We demonstrate the effectiveness of our approach using three metrics on occupancy prediction, motion estimation, and agent ID recovery. In addition, we introduce the problem of predicting speculative agents, which are currently-occluded agents that may appear in the future through dis-occlusion or by entering the field of view. We report experimental results on a large in-house autonomous driving dataset and the public INTERACTION dataset, and show that our model outperforms state-of-the-art models.