# An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds

arXiv: https://arxiv.org/abs/2007.12392 · HTML: https://mlsanity.com/paper/2007.12392/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 30 of 45)

- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [3D-MAN: 3D Multi-frame Attention Network for Object Detection](https://mlsanity.com/paper/2103.16054/) (2021)
- [Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perception](https://mlsanity.com/paper/2303.05970/) (2023)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [4D-Net for Learned Multi-Modal Alignment](https://mlsanity.com/paper/2109.01066/) (2021)
- [To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernels](https://mlsanity.com/paper/2106.13381/) (2021)
- [FWD: Real-time Novel View Synthesis with Forward Warping and Depth](https://mlsanity.com/paper/2206.08355/) (2022)
- [BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregation](https://mlsanity.com/paper/2303.17099/) (2023)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [From Multi-View to Hollow-3D: Hallucinated Hollow-3D R-CNN for 3D Object Detection](https://mlsanity.com/paper/2107.14391/) (2021)
- [Spatial-Temporal Transformer for 3D Point Cloud Sequences](https://mlsanity.com/paper/2110.09783/) (2021)
- [Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception](https://mlsanity.com/paper/2203.11405/) (2022)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection](https://mlsanity.com/paper/2411.03013/) (2024)
- [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022)
- [MGTANet: Encoding Sequential LiDAR Points Using Long Short-Term Motion-Guided Temporal Attention for 3D Object Detection](https://mlsanity.com/paper/2212.00442/) (2022)
- [MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2303.08316/) (2023)
- [INT: Towards Infinite-frames 3D Detection with An Efficient Framework](https://mlsanity.com/paper/2209.15215/) (2022)
- [An Effective Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2303.12535/) (2023)
- [SVQNet: Sparse Voxel-Adjacent Query Network for 4D Spatio-Temporal LiDAR Semantic Segmentation](https://mlsanity.com/paper/2308.13323/) (2023)
- [Semi-supervised 3D Object Detection via Temporal Graph Neural Networks](https://mlsanity.com/paper/2202.00182/) (2022)
- [A Survey of Robust 3D Object Detection Methods in Point Clouds](https://mlsanity.com/paper/2204.00106/) (2022)
- [SpOT: Spatiotemporal Modeling for 3D Object Tracking](https://mlsanity.com/paper/2207.05856/) (2022)
- [An Empirical Analysis of Range for 3D Object Detection](https://mlsanity.com/paper/2308.04054/) (2023)
- [Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2307.00347/) (2023)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [Sequential Point Clouds: A Survey](https://mlsanity.com/paper/2204.09337/) (2022)

## Abstract (arXiv)

Detecting objects in 3D LiDAR data is a core technology for autonomous driving and other robotics applications. Although LiDAR data is acquired over time, most of the 3D object detection algorithms propose object bounding boxes independently for each frame and neglect the useful information available in the temporal domain. To address this problem, in this paper we propose a sparse LSTM-based multi-frame 3d object detection algorithm. We use a U-Net style 3D sparse convolution network to extract features for each frame's LiDAR point-cloud. These features are fed to the LSTM module together with the hidden and memory features from last frame to predict the 3d objects in the current frame as well as hidden and memory features that are passed to the next frame. Experiments on the Waymo Open Dataset show that our algorithm outperforms the traditional frame by frame approach by 7.5% mAP@0.7 and other multi-frame approaches by 1.2% while using less memory and computation per frame. To the best of our knowledge, this is the first work to use an LSTM for 3D object detection in sparse point clouds.