# Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds

arXiv: https://arxiv.org/abs/2207.12659 · HTML: https://mlsanity.com/paper/2207.12659/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 100 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 30 of 84)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)

## Built on by (most cited first, 17 of 17)

- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [SSDA3D: Semi-supervised Domain Adaptation for 3D Object Detection from Point Cloud](https://mlsanity.com/paper/2212.02845/) (2022)
- [Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency](https://mlsanity.com/paper/2303.08686/) (2023)
- [Point Cloud Attacks in Graph Spectral Domain: When 3D Geometry Meets Graph Signal Processing](https://mlsanity.com/paper/2207.13326/) (2022)
- [MGTANet: Encoding Sequential LiDAR Points Using Long Short-Term Motion-Guided Temporal Attention for 3D Object Detection](https://mlsanity.com/paper/2212.00442/) (2022)
- [OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving](https://mlsanity.com/paper/2412.17226/) (2024)
- [LWSIS: LiDAR-guided Weakly Supervised Instance Segmentation for Autonomous Driving](https://mlsanity.com/paper/2212.03504/) (2022)
- [Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysis](https://mlsanity.com/paper/2404.08229/) (2024)
- [Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2307.00347/) (2023)
- [Boosting Monocular 3D Object Detection with Object-Centric Auxiliary Depth Supervision](https://mlsanity.com/paper/2210.16574/) (2022)
- [Fuzzy-NMS: Improving 3D Object Detection with Fuzzy Classification in NMS](https://mlsanity.com/paper/2310.13951/) (2023)
- [Hierarchical Graph Pattern Understanding for Zero-Shot VOS](https://mlsanity.com/paper/2312.09525/) (2023)
- [M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar](https://mlsanity.com/paper/2510.27166/) (2025)
- [Leveraging Multi-stream Information Fusion for Trajectory Prediction in Low-illumination Scenarios: A Multi-channel Graph Convolutional Approach](https://mlsanity.com/paper/2211.10226/) (2022)
- [Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detection](https://mlsanity.com/paper/2503.08992/) (2025)
- [Multimodal Spatio-temporal Graph Learning for Alignment-free RGBT Video Object Detection](https://mlsanity.com/paper/2504.11779/) (2025)
- [PointSlice: Accurate and Efficient Slice-Based Representation for 3D Object Detection from Point Clouds](https://mlsanity.com/paper/2509.01487/) (2025)

## Abstract (arXiv)

Previous works for LiDAR-based 3D object detection mainly focus on the single-frame paradigm. In this paper, we propose to detect 3D objects by exploiting temporal information in multiple frames, i.e., the point cloud videos. We empirically categorize the temporal information into short-term and long-term patterns. To encode the short-term data, we present a Grid Message Passing Network (GMPNet), which considers each grid (i.e., the grouped points) as a node and constructs a k-NN graph with the neighbor grids. To update features for a grid, GMPNet iteratively collects information from its neighbors, thus mining the motion cues in grids from nearby frames. To further aggregate the long-term frames, we propose an Attentive Spatiotemporal Transformer GRU (AST-GRU), which contains a Spatial Transformer Attention (STA) module and a Temporal Transformer Attention (TTA) module. STA and TTA enhance the vanilla GRU to focus on small objects and better align the moving objects. Our overall framework supports both online and offline video object detection in point clouds. We implement our algorithm based on prevalent anchor-based and anchor-free detectors. The evaluation results on the challenging nuScenes benchmark show the superior performance of our method, achieving the 1st on the leaderboard without any bells and whistles, by the time the paper is submitted.