# Towards Category Unification of 3D Single Object Tracking on Point Clouds

arXiv: https://arxiv.org/abs/2401.11204 · HTML: https://mlsanity.com/paper/2401.11204/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 20 Jan 2024.

## Builds on (most cited first, 30 of 39)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)
- [P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds](https://mlsanity.com/paper/2005.13888/) (2020)
- [Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection](https://mlsanity.com/paper/2109.02499/) (2021)
- [Leveraging Shape Completion for 3D Siamese Tracking](https://mlsanity.com/paper/1903.01784/) (2019)
- [Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2108.04728/) (2021)
- [PTTR: Relational 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2112.02857/) (2021)
- [Correlation-Aware Deep Tracking](https://mlsanity.com/paper/2203.01666/) (2022)
- [PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2108.06455/) (2021)
- [CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds](https://mlsanity.com/paper/2210.04264/) (2022)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)
- [Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook](https://mlsanity.com/paper/2112.02838/) (2021)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [RBGNet: Ray-based Grouping for 3D Object Detection](https://mlsanity.com/paper/2204.02251/) (2022)
- [CXTrack: Improving 3D Point Cloud Tracking with Contextual Information](https://mlsanity.com/paper/2211.08542/) (2022)

## Built on by (most cited first, 13 of 13)

- [P2P: Part-to-Part Motion Cues Guide a Strong Tracking Framework for LiDAR Point Clouds](https://mlsanity.com/paper/2407.05238/) (2024)
- [MVCTrack: Boosting 3D Point Cloud Tracking via Multimodal-Guided Virtual Cues](https://mlsanity.com/paper/2412.02734/) (2024)
- [CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking](https://mlsanity.com/paper/2511.15580/) (2025)
- [FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2602.24133/) (2026)
- [VoxelTrack: Exploring Voxel Representation for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2408.02263/) (2024)
- [TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking](https://mlsanity.com/paper/2507.19908/) (2025)
- [PillarTrack:Boosting Pillar Representation for Transformer-based 3D Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2404.07495/) (2024)
- [GSOT3D: Towards Generic 3D Single Object Tracking in the Wild](https://mlsanity.com/paper/2412.02129/) (2024)
- [Visual Object Tracking across Diverse Data Modalities: A Review](https://mlsanity.com/paper/2412.09991/) (2024)
- [Mamba-Adaptor: State Space Model Adaptor for Visual Recognition](https://mlsanity.com/paper/2505.12685/) (2025)
- [TFTrack: A Template-Free Framework for Efficient 3D Point Cloud Tracking](https://mlsanity.com/paper/2609.07738/) (2026)
- [GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing](https://mlsanity.com/paper/2602.08550/) (2026)
- [SAVTrack: Selective Vote Aggregation for Reliability-Aware Point Cloud Tracking](https://mlsanity.com/paper/2609.16662/) (2026)

## Abstract (arXiv)

Category-specific models are provenly valuable methods in 3D single object tracking (SOT) regardless of Siamese or motion-centric paradigms. However, such over-specialized model designs incur redundant parameters, thus limiting the broader applicability of 3D SOT task. This paper first introduces unified models that can simultaneously track objects across all categories using a single network with shared model parameters. Specifically, we propose to explicitly encode distinct attributes associated to different object categories, enabling the model to adapt to cross-category data. We find that the attribute variances of point cloud objects primarily occur from the varying size and shape (e.g., large and square vehicles v.s. small and slender humans). Based on this observation, we design a novel point set representation learning network inheriting transformer architecture, termed AdaFormer, which adaptively encodes the dynamically varying shape and size information from cross-category data in a unified manner. We further incorporate the size and shape prior derived from the known template targets into the model's inputs and learning objective, facilitating the learning of unified representation. Equipped with such designs, we construct two category-unified models SiamCUT and MoCUT.Extensive experiments demonstrate that SiamCUT and MoCUT exhibit strong generalization and training stability. Furthermore, our category-unified models outperform the category-specific counterparts by a significant margin (e.g., on KITTI dataset, 12% and 3% performance gains on the Siamese and motion paradigms). Our code will be available.