# Integrating Specialized and Generic Agent Motion Prediction with Dynamic Occupancy Grid Maps

arXiv: https://arxiv.org/abs/2602.07938 · HTML: https://mlsanity.com/paper/2602.07938/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 30 of 30)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Occupancy Flow Fields for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2203.03875/) (2022)
- [FISHING Net: Future Inference of Semantic Heatmaps In Grids](https://mlsanity.com/paper/2006.09917/) (2020)
- [Probabilistic Future Prediction for Video Scene Understanding](https://mlsanity.com/paper/2003.06409/) (2020)
- [Long-Term Occupancy Grid Prediction Using Recurrent Neural Networks](https://mlsanity.com/paper/1809.03782/) (2018)
- [Multi-Step Prediction of Occupancy Grid Maps with Recurrent Neural Networks](https://mlsanity.com/paper/1812.09395/) (2018)
- [CRAT-Pred: Vehicle Trajectory Prediction with Crystal Graph Convolutional Neural Networks and Multi-Head Self-Attention](https://mlsanity.com/paper/2202.04488/) (2022)
- [Dynamic Environment Prediction in Urban Scenes using Recurrent Representation Learning](https://mlsanity.com/paper/1904.12374/) (2019)
- [Dynamics-Aware Spatiotemporal Occupancy Prediction in Urban Environments](https://mlsanity.com/paper/2209.13172/) (2022)
- [Localization Requirements for Autonomous Vehicles](https://mlsanity.com/paper/1906.01061/) (2019)
- [Attention Augmented ConvLSTM for Environment Prediction](https://mlsanity.com/paper/2010.09662/) (2020)
- [Physically Feasible Vehicle Trajectory Prediction](https://mlsanity.com/paper/2104.14679/) (2021)
- [Context-Aware Scene Prediction Network (CASPNet)](https://mlsanity.com/paper/2201.06933/) (2022)
- [Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learning](https://mlsanity.com/paper/2205.03212/) (2022)
- [Predicting Future Spatiotemporal Occupancy Grids with Semantics for Autonomous Driving](https://mlsanity.com/paper/2310.01723/) (2023)
- [Allo-centric Occupancy Grid Prediction for Urban Traffic Scene Using Video Prediction Networks](https://mlsanity.com/paper/2301.04454/) (2023)
- [Vehicle Motion Forecasting using Prior Information and Semantic-assisted Occupancy Grid Maps](https://mlsanity.com/paper/2308.04303/) (2023)
- [Flow-guided Motion Prediction with Semantics and Dynamic Occupancy Grid Maps](https://mlsanity.com/paper/2407.15675/) (2024)

## Abstract (arXiv)

Accurate prediction of driving scene is a challenging task due to uncertainty in sensor data, the complex behaviors of agents, and the possibility of multiple feasible futures. Existing prediction methods using occupancy grid maps primarily focus on agent-agnostic scene predictions, while agent-specific predictions provide specialized behavior insights with the help of semantic information. However, both paradigms face distinct limitations: agent-agnostic models struggle to capture the behavioral complexities of dynamic actors, whereas agent-specific approaches fail to generalize to poorly perceived or unrecognized agents; combining both enables robust and safer motion forecasting. To address this, we propose a unified framework by leveraging Dynamic Occupancy Grid Maps within a streamlined temporal decoding pipeline to simultaneously predict future occupancy state grids, vehicle grids, and scene flow grids. Relying on a lightweight spatiotemporal backbone, our approach is centered on a tailored, interdependent loss function that captures inter-grid dependencies and enables diverse future predictions. By using occupancy state information to enforce flow-guided transitions, the loss function acts as a regularizer that directs occupancy evolution while accounting for obstacles and occlusions. Consequently, the model not only predicts the specific behaviors of vehicle agents, but also identifies other dynamic entities and anticipates their evolution within the complex scene. Evaluations on real-world nuScenes and Woven Planet datasets demonstrate superior prediction performances for dynamic vehicles and generic dynamic scene elements compared to baseline methods.