# 3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud

arXiv: https://arxiv.org/abs/2108.05630 · HTML: https://mlsanity.com/paper/2108.05630/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [End-to-end representation learning for Correlation Filter based tracking](https://mlsanity.com/paper/1704.06036/) (2017)
- [P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds](https://mlsanity.com/paper/2005.13888/) (2020)
- [The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes](https://mlsanity.com/paper/1903.01568/) (2019)
- [Joint Monocular 3D Vehicle Detection and Tracking](https://mlsanity.com/paper/1811.10742/) (2018)
- [Leveraging Shape Completion for 3D Siamese Tracking](https://mlsanity.com/paper/1903.01784/) (2019)
- [Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking](https://mlsanity.com/paper/1608.03773/) (2016)
- [Staple: Complementary Learners for Real-Time Tracking](https://mlsanity.com/paper/1512.01355/) (2015)
- [Complex-YOLO: Real-time 3D Object Detection on Point Clouds](https://mlsanity.com/paper/1803.06199/) (2018)
- [JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset](https://mlsanity.com/paper/2002.08397/) (2020)
- [Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds](https://mlsanity.com/paper/1904.07537/) (2019)
- [Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering](https://mlsanity.com/paper/1802.09975/) (2018)
- [FANTrack: 3D Multi-Object Tracking with Feature Association Network](https://mlsanity.com/paper/1905.02843/) (2019)
- [End-to-end Learning of Multi-sensor 3D Tracking by Detection](https://mlsanity.com/paper/1806.11534/) (2018)
- [PointTrackNet: An End-to-End Network For 3-D Object Detection and Tracking From Point Clouds](https://mlsanity.com/paper/2002.11559/) (2020)

## Built on by (most cited first, 30 of 45)

- [PTTR: Relational 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2112.02857/) (2021)
- [PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2108.06455/) (2021)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [FastPillars: A Deployment-friendly Pillar-based 3D Detector](https://mlsanity.com/paper/2302.02367/) (2023)
- [CXTrack: Improving 3D Point Cloud Tracking with Contextual Information](https://mlsanity.com/paper/2211.08542/) (2022)
- [Real-time 3D Single Object Tracking with Transformer](https://mlsanity.com/paper/2209.00860/) (2022)
- [MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors](https://mlsanity.com/paper/2303.05071/) (2023)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [GLT-T: Global-Local Transformer Voting for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2211.10927/) (2022)
- [Exploring Point-BEV Fusion for 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2208.05216/) (2022)
- [OSP2B: One-Stage Point-to-Box Network for 3D Siamese Tracking](https://mlsanity.com/paper/2304.11584/) (2023)
- [Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking](https://mlsanity.com/paper/2308.12549/) (2023)
- [An Effective Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2303.12535/) (2023)
- [Information Entropy Guided Height-aware Histogram for Quantization-friendly Pillar Feature Encoder](https://mlsanity.com/paper/2405.18734/) (2024)
- [Correlation Pyramid Network for 3D Single Object Tracking](https://mlsanity.com/paper/2305.09195/) (2023)
- [P2P: Part-to-Part Motion Cues Guide a Strong Tracking Framework for LiDAR Point Clouds](https://mlsanity.com/paper/2407.05238/) (2024)
- [MVCTrack: Boosting 3D Point Cloud Tracking via Multimodal-Guided Virtual Cues](https://mlsanity.com/paper/2412.02734/) (2024)
- [CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking](https://mlsanity.com/paper/2511.15580/) (2025)
- [Modeling Continuous Motion for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2303.07605/) (2023)
- [Robust 3D Tracking with Quality-Aware Shape Completion](https://mlsanity.com/paper/2312.10608/) (2023)
- [FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2602.24133/) (2026)
- [Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild](https://mlsanity.com/paper/2111.12728/) (2021)
- [Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking](https://mlsanity.com/paper/2209.00522/) (2022)
- [Towards Class-agnostic Tracking Using Feature Decorrelation in Point Clouds](https://mlsanity.com/paper/2202.13524/) (2022)
- [GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss](https://mlsanity.com/paper/2304.00242/) (2023)
- [3D Single-object Tracking in Point Clouds with High Temporal Variation](https://mlsanity.com/paper/2408.02049/) (2024)
- [TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking](https://mlsanity.com/paper/2507.19908/) (2025)
- [Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking](https://mlsanity.com/paper/2509.11453/) (2025)
- [VPIT: Real-time Embedded Single Object 3D Tracking Using Voxel Pseudo Images](https://mlsanity.com/paper/2206.02619/) (2022)

## Abstract (arXiv)

3D single object tracking is a key issue for autonomous following robot, where the robot should robustly track and accurately localize the target for efficient following. In this paper, we propose a 3D tracking method called 3D-SiamRPN Network to track a single target object by using raw 3D point cloud data. The proposed network consists of two subnetworks. The first subnetwork is feature embedding subnetwork which is used for point cloud feature extraction and fusion. In this subnetwork, we first use PointNet++ to extract features of point cloud from template and search branches. Then, to fuse the information of features in the two branches and obtain their similarity, we propose two cross correlation modules, named Pointcloud-wise and Point-wise respectively. The second subnetwork is region proposal network(RPN), which is used to get the final 3D bounding box of the target object based on the fusion feature from cross correlation modules. In this subnetwork, we utilize the regression and classification branches of a region proposal subnetwork to obtain proposals and scores, thus get the final 3D bounding box of the target object. Experimental results on KITTI dataset show that our method has a competitive performance in both Success and Precision compared to the state-of-the-art methods, and could run in real-time at 20.8 FPS. Additionally, experimental results on H3D dataset demonstrate that our method also has good generalization ability and could achieve good tracking performance in a new scene without re-training.