# 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation

arXiv: https://arxiv.org/abs/2110.02531 · HTML: https://mlsanity.com/paper/2110.02531/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 30 of 33)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Flow-Guided Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/1703.10025/) (2017)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [Memory Enhanced Global-Local Aggregation for Video Object Detection](https://mlsanity.com/paper/2003.12063/) (2020)
- [Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection](https://mlsanity.com/paper/1903.01864/) (2019)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [Learning to Track at 100 FPS with Deep Regression Networks](https://mlsanity.com/paper/1604.01802/) (2016)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [What You See is What You Get: Exploiting Visibility for 3D Object Detection](https://mlsanity.com/paper/1912.04986/) (2019)
- [Seq-NMS for Video Object Detection](https://mlsanity.com/paper/1602.08465/) (2016)
- [LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention](https://mlsanity.com/paper/2004.01389/) (2020)
- [Track to Reconstruct and Reconstruct to Track](https://mlsanity.com/paper/1910.00130/) (2019)
- [StarNet: Targeted Computation for Object Detection in Point Clouds](https://mlsanity.com/paper/1908.11069/) (2019)
- [An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2007.12392/) (2020)
- [Improving 3D Object Detection through Progressive Population Based Augmentation](https://mlsanity.com/paper/2004.00831/) (2020)

## Built on by (most cited first, 1 of 1)

- [Minuet: Accelerating 3D Sparse Convolutions on GPUs](https://mlsanity.com/paper/2401.06145/) (2023)

## Abstract (arXiv)

3D object detection using LiDAR data remains a key task for applications like autonomous driving and robotics. Unlike in the case of 2D images, LiDAR data is almost always collected over a period of time. However, most work in this area has focused on performing detection independent of the temporal domain. In this paper we present 3D-FCT, a Siamese network architecture that utilizes temporal information to simultaneously perform the related tasks of 3D object detection and tracking. The network is trained to predict the movement of an object based on the correlation features of extracted keypoints across time. Calculating correlation across keypoints only allows for real-time object detection. We further extend the multi-task objective to include a tracking regression loss. Finally, we produce high accuracy detections by linking short-term object tracklets into long term tracks based on the predicted tracks. Our proposed method is evaluated on the KITTI tracking dataset where it is shown to provide an improvement of 5.57% mAP over a state-of-the-art approach.