# Spatio-Temporal Bi-directional Cross-frame Memory for Distractor Filtering Point Cloud Single Object Tracking

arXiv: https://arxiv.org/abs/2403.15831 · HTML: https://mlsanity.com/paper/2403.15831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds](https://mlsanity.com/paper/2005.13888/) (2020)
- [Leveraging Shape Completion for 3D Siamese Tracking](https://mlsanity.com/paper/1903.01784/) (2019)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2108.04728/) (2021)
- [PTTR: Relational 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2112.02857/) (2021)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)
- [GiT: Graph Interactive Transformer for Vehicle Re-identification](https://mlsanity.com/paper/2107.05475/) (2021)
- [3D Object Tracking with Transformer](https://mlsanity.com/paper/2110.14921/) (2021)
- [3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud](https://mlsanity.com/paper/2108.05630/) (2021)
- [VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentation](https://mlsanity.com/paper/1904.08963/) (2019)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [Exploring Spatial Significance via Hybrid Pyramidal Graph Network for Vehicle Re-identification](https://mlsanity.com/paper/2005.14684/) (2020)
- [CXTrack: Improving 3D Point Cloud Tracking with Contextual Information](https://mlsanity.com/paper/2211.08542/) (2022)
- [Real-time 3D Single Object Tracking with Transformer](https://mlsanity.com/paper/2209.00860/) (2022)
- [Model-free Vehicle Tracking and State Estimation in Point Cloud Sequences](https://mlsanity.com/paper/2103.06028/) (2021)
- [MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors](https://mlsanity.com/paper/2303.05071/) (2023)
- [GLT-T: Global-Local Transformer Voting for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2211.10927/) (2022)
- [Learning to Filter: Siamese Relation Network for Robust Tracking](https://mlsanity.com/paper/2104.00829/) (2021)
- [Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking](https://mlsanity.com/paper/2210.00519/) (2022)
- [Distractor-Aware Fast Tracking via Dynamic Convolutions and MOT Philosophy](https://mlsanity.com/paper/2104.12041/) (2021)
- [STTracker: Spatio-Temporal Tracker for 3D Single Object Tracking](https://mlsanity.com/paper/2306.17440/) (2023)
- [Modeling Continuous Motion for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2303.07605/) (2023)
- [Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking](https://mlsanity.com/paper/2209.00522/) (2022)

## Abstract (arXiv)

3D single object tracking within LIDAR point clouds is a pivotal task in computer vision, with profound implications for autonomous driving and robotics. However, existing methods, which depend solely on appearance matching via Siamese networks or utilize motion information from successive frames, encounter significant challenges. Issues such as similar objects nearby or occlusions can result in tracker drift. To mitigate these challenges, we design an innovative spatio-temporal bi-directional cross-frame distractor filtering tracker, named STMD-Tracker. Our first step involves the creation of a 4D multi-frame spatio-temporal graph convolution backbone. This design separates KNN graph spatial embedding and incorporates 1D temporal convolution, effectively capturing temporal fluctuations and spatio-temporal information. Subsequently, we devise a novel bi-directional cross-frame memory procedure. This integrates future and synthetic past frame memory to enhance the current memory, thereby improving the accuracy of iteration-based tracking. This iterative memory update mechanism allows our tracker to dynamically compensate for information in the current frame, effectively reducing tracker drift. Lastly, we construct spatially reliable Gaussian masks on the fused features to eliminate distractor points. This is further supplemented by an object-aware sampling strategy, which bolsters the efficiency and precision of object localization, thereby reducing tracking errors caused by distractors. Our extensive experiments on KITTI, NuScenes and Waymo datasets demonstrate that our approach significantly surpasses the current state-of-the-art methods.