# Detect to Track and Track to Detect

arXiv: https://arxiv.org/abs/1710.03958 · HTML: https://mlsanity.com/paper/1710.03958/

## Key facts

- Cited by 131 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2017.

## Built on by (most cited first, 30 of 131)

- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [MeMOT: Multi-Object Tracking with Memory](https://mlsanity.com/paper/2203.16761/) (2022)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [Multiple Object Tracking with Correlation Learning](https://mlsanity.com/paper/2104.03541/) (2021)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking](https://mlsanity.com/paper/2210.06984/) (2022)
- [DEFT: Detection Embeddings for Tracking](https://mlsanity.com/paper/2102.02267/) (2021)
- [Exploring Simple 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2108.10312/) (2021)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds](https://mlsanity.com/paper/2111.04426/) (2021)
- [OVTrack: Open-Vocabulary Multiple Object Tracking](https://mlsanity.com/paper/2304.08408/) (2023)
- [RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation](https://mlsanity.com/paper/2105.04322/) (2021)
- [SparseTrack: Multi-Object Tracking by Performing Scene Decomposition based on Pseudo-Depth](https://mlsanity.com/paper/2306.05238/) (2023)
- [GIAOTracker: A comprehensive framework for MCMOT with global information and optimizing strategies in VisDrone 2021](https://mlsanity.com/paper/2202.11983/) (2022)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation](https://mlsanity.com/paper/2312.08952/) (2023)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [Tracking Pedestrian Heads in Dense Crowd](https://mlsanity.com/paper/2103.13516/) (2021)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking](https://mlsanity.com/paper/2305.14298/) (2023)
- [End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2105.10920/) (2021)
- [TF-Blender: Temporal Feature Blender for Video Object Detection](https://mlsanity.com/paper/2108.05821/) (2021)
- [Tracking Objects as Pixel-wise Distributions](https://mlsanity.com/paper/2207.05518/) (2022)
- [Deep Learning for UAV-based Object Detection and Tracking: A Survey](https://mlsanity.com/paper/2110.12638/) (2021)
- [Towards Discriminative Representation: Multi-view Trajectory Contrastive Learning for Online Multi-object Tracking](https://mlsanity.com/paper/2203.14208/) (2022)

## Abstract (arXiv)

Recent approaches for high accuracy detection and tracking of object categories in video consist of complex multistage solutions that become more cumbersome each year. In this paper we propose a ConvNet architecture that jointly performs detection and tracking, solving the task in a simple and effective way. Our contributions are threefold: (i) we set up a ConvNet architecture for simultaneous detection and tracking, using a multi-task objective for frame-based object detection and across-frame track regression; (ii) we introduce correlation features that represent object co-occurrences across time to aid the ConvNet during tracking; and (iii) we link the frame level detections based on our across-frame tracklets to produce high accuracy detections at the video level. Our ConvNet architecture for spatiotemporal object detection is evaluated on the large-scale ImageNet VID dataset where it achieves state-of-the-art results. Our approach provides better single model performance than the winning method of the last ImageNet challenge while being conceptually much simpler. Finally, we show that by increasing the temporal stride we can dramatically increase the tracker speed.