# Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking

arXiv: https://arxiv.org/abs/2106.16100 · HTML: https://mlsanity.com/paper/2106.16100/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identification](https://mlsanity.com/paper/1711.07027/) (2017)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Dissecting Person Re-identification from the Viewpoint of Viewpoint](https://mlsanity.com/paper/1812.02162/) (2018)
- [Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World](https://mlsanity.com/paper/1803.08319/) (2018)
- [Online Multi-Object Tracking with Dual Matching Attention Networks](https://mlsanity.com/paper/1902.00749/) (2019)
- [Sim2real transfer learning for 3D human pose estimation: motion to the rescue](https://mlsanity.com/paper/1907.02499/) (2019)
- [How To Train Your Deep Multi-Object Tracker](https://mlsanity.com/paper/1906.06618/) (2019)
- [Multiview Detection with Feature Perspective Transformation](https://mlsanity.com/paper/2007.07247/) (2020)
- [Domain Adaptation through Synthesis for Unsupervised Person Re-identification](https://mlsanity.com/paper/1804.10094/) (2018)
- [Simulating Content Consistent Vehicle Datasets with Attribute Descent](https://mlsanity.com/paper/1912.08855/) (2019)
- [GCNNMatch: Graph Convolutional Neural Networks for Multi-Object Tracking via Sinkhorn Normalization](https://mlsanity.com/paper/2010.00067/) (2020)
- [VersatileGait: A Large-Scale Synthetic Gait Dataset with Fine-GrainedAttributes and Complicated Scenarios](https://mlsanity.com/paper/2101.01394/) (2021)
- [Learning to simulate complex scenes](https://mlsanity.com/paper/2006.14611/) (2020)

## Built on by (most cited first, 1 of 1)

- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)

## Abstract (arXiv)

Association, aiming to link bounding boxes of the same identity in a video sequence, is a central component in multi-object tracking (MOT). To train association modules, e.g., parametric networks, real video data are usually used. However, annotating person tracks in consecutive video frames is expensive, and such real data, due to its inflexibility, offer us limited opportunities to evaluate the system performance w.r.t changing tracking scenarios. In this paper, we study whether 3D synthetic data can replace real-world videos for association training. Specifically, we introduce a large-scale synthetic data engine named MOTX, where the motion characteristics of cameras and objects are manually configured to be similar to those in real-world datasets. We show that compared with real data, association knowledge obtained from synthetic data can achieve very similar performance on real-world test sets without domain adaption techniques. Our intriguing observation is credited to two factors. First and foremost, 3D engines can well simulate motion factors such as camera movement, camera view and object movement, so that the simulated videos can provide association modules with effective motion features. Second, experimental results show that the appearance domain gap hardly harms the learning of association knowledge. In addition, the strong customization ability of MOTX allows us to quantitatively assess the impact of motion factors on MOT, which brings new insights to the community.