# MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences

arXiv: https://arxiv.org/abs/2303.08316 · HTML: https://mlsanity.com/paper/2303.08316/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [RangeDet:In Defense of Range View for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2103.10039/) (2021)
- [Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds](https://mlsanity.com/paper/2203.11139/) (2022)
- [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](https://mlsanity.com/paper/2103.15297/) (2021)
- [RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection](https://mlsanity.com/paper/2106.13365/) (2021)
- [Improving 3D Object Detection with Channel-wise Transformer](https://mlsanity.com/paper/2108.10723/) (2021)
- [Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds](https://mlsanity.com/paper/2203.10314/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2210.07372/) (2022)
- [M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers](https://mlsanity.com/paper/2104.11896/) (2021)
- [3D-MAN: 3D Multi-frame Attention Network for Object Detection](https://mlsanity.com/paper/2103.16054/) (2021)
- [LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention](https://mlsanity.com/paper/2004.01389/) (2020)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [StarNet: Targeted Computation for Object Detection in Point Clouds](https://mlsanity.com/paper/1908.11069/) (2019)
- [An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2007.12392/) (2020)

## Built on by (most cited first, 15 of 15)

- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024)
- [ScatterFormer: Efficient Voxel Transformer with Scattered Linear Attention](https://mlsanity.com/paper/2401.00912/) (2024)
- [PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes](https://mlsanity.com/paper/2309.10815/) (2023)
- [Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2307.00347/) (2023)
- [PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection](https://mlsanity.com/paper/2312.08371/) (2023)
- [General Geometry-aware Weakly Supervised 3D Object Detection](https://mlsanity.com/paper/2407.13748/) (2024)
- [Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation](https://mlsanity.com/paper/2409.11018/) (2024)
- [GraphRelate3D: Context-Dependent 3D Object Detection with Inter-Object Relationship Graphs](https://mlsanity.com/paper/2405.06782/) (2024)
- [Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences](https://mlsanity.com/paper/2409.04390/) (2024)
- [TrajSSL: Trajectory-Enhanced Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2409.10901/) (2024)
- [FASTer: Focal Token Acquiring-and-Scaling Transformer for Long-term 3D Object Detection](https://mlsanity.com/paper/2503.01899/) (2025)
- [STELLAR: Scaling 3D Perception Large Models for Autonomous Driving](https://mlsanity.com/paper/2605.20390/) (2026)
- [Scene Reconstruction as Mapping Priors for 3D Detection](https://mlsanity.com/paper/2605.22997/) (2026)
- [ATLAS: A Large-Scale Evaluation Benchmark for Adversarial LiDAR Perception](https://mlsanity.com/paper/2606.02924/) (2026)

## Abstract (arXiv)

Point cloud sequences are commonly used to accurately detect 3D objects in applications such as autonomous driving. Current top-performing multi-frame detectors mostly follow a Detect-and-Fuse framework, which extracts features from each frame of the sequence and fuses them to detect the objects in the current frame. However, this inevitably leads to redundant computation since adjacent frames are highly correlated. In this paper, we propose an efficient Motion-guided Sequential Fusion (MSF) method, which exploits the continuity of object motion to mine useful sequential contexts for object detection in the current frame. We first generate 3D proposals on the current frame and propagate them to preceding frames based on the estimated velocities. The points-of-interest are then pooled from the sequence and encoded as proposal features. A novel Bidirectional Feature Aggregation (BiFA) module is further proposed to facilitate the interactions of proposal features across frames. Besides, we optimize the point cloud pooling by a voxel-based sampling technique so that millions of points can be processed in several milliseconds. The proposed MSF method achieves not only better efficiency than other multi-frame detectors but also leading accuracy, with 83.12% and 78.30% mAP on the LEVEL1 and LEVEL2 test sets of Waymo Open Dataset, respectively. Codes can be found at urlhttps://github.com/skyhehe123/MSF.