# SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking

arXiv: https://arxiv.org/abs/2411.06780 · HTML: https://mlsanity.com/paper/2411.06780/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection](https://mlsanity.com/paper/1904.09730/) (2019)
- [MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors](https://mlsanity.com/paper/2211.09791/) (2022)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries](https://mlsanity.com/paper/2205.00613/) (2022)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking](https://mlsanity.com/paper/2305.14298/) (2023)
- [NMS Strikes Back](https://mlsanity.com/paper/2212.06137/) (2022)
- [ADA-Track++: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association](https://mlsanity.com/paper/2405.08909/) (2024)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)

## Built on by (most cited first, 2 of 2)

- [TQD-Track: Temporal Query Denoising for 3D Multi-Object Tracking](https://mlsanity.com/paper/2504.03258/) (2025)
- [Fusion-Poly: A Polyhedral Framework Based on Spatial-Temporal Fusion for 3D Multi-Object Tracking](https://mlsanity.com/paper/2603.08199/) (2026)

## Abstract (arXiv)

While existing query-based 3D end-to-end visual trackers integrate detection and tracking via the tracking-by-attention paradigm, these two chicken-and-egg tasks encounter optimization difficulties when sharing the same parameters. Our findings reveal that these difficulties arise due to two inherent constraints on the self-attention mechanism, i.e., over-deduplication for object queries and self-centric attention for track queries. In contrast, removing the self-attention mechanism not only minimally impacts regression predictions of the tracker, but also tends to generate more latent candidate boxes. Based on these analyses, we present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. Specifically, we propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder that matches the targets of track queries with multiple object queries to exploit promising candidates overlooked by the self-attention mechanism. To flexibly select optimal candidates for the one-to-many matching, we also design a Dynamic Query Filtering module controlled by model training status. Moreover, we introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries, effectively bridging the gap between detection and tracking. Without additional inference costs, SynCL consistently delivers improvements in various benchmarks and achieves state-of-the-art performance with 58.9% AMOTA on the nuScenes dataset. Code and raw results will be publicly available.