# MoDAR: Using Motion Forecasting for 3D Object Detection in Point Cloud Sequences

arXiv: https://arxiv.org/abs/2306.03206 · HTML: https://mlsanity.com/paper/2306.03206/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 64 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 30 of 56)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Multimodal Virtual Point 3D Detection](https://mlsanity.com/paper/2111.06881/) (2021)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)

## Built on by (most cited first, 13 of 13)

- [T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory](https://mlsanity.com/paper/2403.10052/) (2024)
- [PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection](https://mlsanity.com/paper/2312.08371/) (2023)
- [Practical Collaborative Perception: A Framework for Asynchronous and Multi-Agent 3D Object Detection](https://mlsanity.com/paper/2307.01462/) (2023)
- [ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2508.07089/) (2025)
- [Frame Fusion with Vehicle Motion Prediction for 3D Object Detection](https://mlsanity.com/paper/2306.10699/) (2023)
- [Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences](https://mlsanity.com/paper/2409.04390/) (2024)
- [Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection](https://mlsanity.com/paper/2412.05154/) (2024)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [Is Intermediate Fusion All You Need for UAV-based Collaborative Perception?](https://mlsanity.com/paper/2504.21774/) (2025)
- [TrajSSL: Trajectory-Enhanced Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2409.10901/) (2024)
- [STELLAR: Scaling 3D Perception Large Models for Autonomous Driving](https://mlsanity.com/paper/2605.20390/) (2026)
- [Scene Reconstruction as Mapping Priors for 3D Detection](https://mlsanity.com/paper/2605.22997/) (2026)
- [CERF: Communication-Efficient and Retraining-Free Collaborative Perception](https://mlsanity.com/paper/2609.00951/) (2026)

## Abstract (arXiv)

Occluded and long-range objects are ubiquitous and challenging for 3D object detection. Point cloud sequence data provide unique opportunities to improve such cases, as an occluded or distant object can be observed from different viewpoints or gets better visibility over time. However, the efficiency and effectiveness in encoding long-term sequence data can still be improved. In this work, we propose MoDAR, using motion forecasting outputs as a type of virtual modality, to augment LiDAR point clouds. The MoDAR modality propagates object information from temporal contexts to a target frame, represented as a set of virtual points, one for each object from a waypoint on a forecasted trajectory. A fused point cloud of both raw sensor points and the virtual points can then be fed to any off-the-shelf point-cloud based 3D object detector. Evaluated on the Waymo Open Dataset, our method significantly improves prior art detectors by using motion forecasting from extra-long sequences (e.g. 18 seconds), achieving new state of the arts, while not adding much computation overhead.