# Multi-Correlation Siamese Transformer Network with Dense Connection for 3D Single Object Tracking

arXiv: https://arxiv.org/abs/2312.11051 · HTML: https://mlsanity.com/paper/2312.11051/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [SwinTrack: A Simple and Strong Baseline for Transformer Tracking](https://mlsanity.com/paper/2112.00995/) (2021)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds](https://mlsanity.com/paper/2005.13888/) (2020)
- [Leveraging Shape Completion for 3D Siamese Tracking](https://mlsanity.com/paper/1903.01784/) (2019)
- [AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2201.06493/) (2022)
- [Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2108.04728/) (2021)
- [PTTR: Relational 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2112.02857/) (2021)
- [PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2108.06455/) (2021)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)

## Built on by (most cited first, 1 of 1)

- [Spatio-Temporal Bi-directional Cross-frame Memory for Distractor Filtering Point Cloud Single Object Tracking](https://mlsanity.com/paper/2403.15831/) (2024)

## Abstract (arXiv)

Point cloud-based 3D object tracking is an important task in autonomous driving. Though great advances regarding Siamese-based 3D tracking have been made recently, it remains challenging to learn the correlation between the template and search branches effectively with the sparse LIDAR point cloud data. Instead of performing correlation of the two branches at just one point in the network, in this paper, we present a multi-correlation Siamese Transformer network that has multiple stages and carries out feature correlation at the end of each stage based on sparse pillars. More specifically, in each stage, self-attention is first applied to each branch separately to capture the non-local context information. Then, cross-attention is used to inject the template information into the search area. This strategy allows the feature learning of the search area to be aware of the template while keeping the individual characteristics of the template intact. To enable the network to easily preserve the information learned at different stages and ease the optimization, for the search area, we densely connect the initial input sparse pillars and the output of each stage to all subsequent stages and the target localization network, which converts pillars to bird's eye view (BEV) feature maps and predicts the state of the target with a small densely connected convolution network. Deep supervision is added to each stage to further boost the performance as well. The proposed algorithm is evaluated on the popular KITTI, nuScenes, and Waymo datasets, and the experimental results show that our method achieves promising performance compared with the state-of-the-art. Ablation study that shows the effectiveness of each component is provided as well. Code is available at https://github.com/liangp/MCSTN-3DSOT.