# LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention

arXiv: https://arxiv.org/abs/2004.01389 · HTML: https://mlsanity.com/paper/2004.01389/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2020.

## Built on by (most cited first, 30 of 65)

- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Multimodal Virtual Point 3D Detection](https://mlsanity.com/paper/2111.06881/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark](https://mlsanity.com/paper/2203.11089/) (2022)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection](https://mlsanity.com/paper/2205.07403/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection](https://mlsanity.com/paper/2106.12449/) (2021)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds](https://mlsanity.com/paper/2112.09205/) (2021)
- [3D-MAN: 3D Multi-frame Attention Network for Object Detection](https://mlsanity.com/paper/2103.16054/) (2021)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [Semi-supervised 3D Object Detection with Proficient Teachers](https://mlsanity.com/paper/2207.12655/) (2022)
- [ProposalContrast: Unsupervised Pre-training for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2207.12654/) (2022)
- [Graph R-CNN: Towards Accurate 3D Object Detection with Semantic-Decorated Local Graph](https://mlsanity.com/paper/2208.03624/) (2022)
- [Delving into Sequential Patches for Deepfake Detection](https://mlsanity.com/paper/2207.02803/) (2022)
- [FastPillars: A Deployment-friendly Pillar-based 3D Detector](https://mlsanity.com/paper/2302.02367/) (2023)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment](https://mlsanity.com/paper/2103.02263/) (2021)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [Spatial-Temporal Transformer for 3D Point Cloud Sequences](https://mlsanity.com/paper/2110.09783/) (2021)
- [Boosting 3D Object Detection by Simulating Multimodality on Point Clouds](https://mlsanity.com/paper/2206.14971/) (2022)
- [PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2305.04075/) (2023)
- [Exploiting Playbacks in Unsupervised Domain Adaptation for 3D Object Detection](https://mlsanity.com/paper/2103.14198/) (2021)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency](https://mlsanity.com/paper/2303.08686/) (2023)
- [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022)
- [IAFA: Instance-aware Feature Aggregation for 3D Object Detection from a Single Image](https://mlsanity.com/paper/2103.03480/) (2021)

## Abstract (arXiv)

Existing LiDAR-based 3D object detectors usually focus on the single-frame detection, while ignoring the spatiotemporal information in consecutive point cloud frames. In this paper, we propose an end-to-end online 3D video object detector that operates on point cloud sequences. The proposed model comprises a spatial feature encoding component and a spatiotemporal feature aggregation component. In the former component, a novel Pillar Message Passing Network (PMPNet) is proposed to encode each discrete point cloud frame. It adaptively collects information for a pillar node from its neighbors by iterative message passing, which effectively enlarges the receptive field of the pillar feature. In the latter component, we propose an Attentive Spatiotemporal Transformer GRU (AST-GRU) to aggregate the spatiotemporal information, which enhances the conventional ConvGRU with an attentive memory gating mechanism. AST-GRU contains a Spatial Transformer Attention (STA) module and a Temporal Transformer Attention (TTA) module, which can emphasize the foreground objects and align the dynamic objects, respectively. Experimental results demonstrate that the proposed 3D video object detector achieves state-of-the-art performance on the large-scale nuScenes benchmark.