# Is Synthetic Dataset Reliable for Benchmarking Generalizable Person Re-Identification?

arXiv: https://arxiv.org/abs/2209.05047 · HTML: https://mlsanity.com/paper/2209.05047/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [Omni-Scale Feature Learning for Person Re-Identification](https://mlsanity.com/paper/1905.00953/) (2019)
- [Learning Discriminative Features with Multiple Granularities for Person Re-Identification](https://mlsanity.com/paper/1804.01438/) (2018)
- [Re-ranking Person Re-identification with k-reciprocal Encoding](https://mlsanity.com/paper/1701.08398/) (2017)
- [FastReID: A Pytorch Toolbox for General Instance Re-identification](https://mlsanity.com/paper/2006.02631/) (2020)
- [Dissecting Person Re-identification from the Viewpoint of Viewpoint](https://mlsanity.com/paper/1812.02162/) (2018)
- [Learning Generalisable Omni-Scale Representations for Person Re-Identification](https://mlsanity.com/paper/1910.06827/) (2019)
- [Multi-Level Factorisation Net for Person Re-Identification](https://mlsanity.com/paper/1803.09132/) (2018)
- [Frustratingly Easy Person Re-Identification: Generalizing Person Re-ID in Practice](https://mlsanity.com/paper/1905.03422/) (2019)
- [Domain Adaptation through Synthesis for Unsupervised Person Re-identification](https://mlsanity.com/paper/1804.10094/) (2018)
- [Surpassing Real-World Source Training Data: Random 3D Characters for Generalizable Person Re-Identification](https://mlsanity.com/paper/2006.12774/) (2020)
- [Torchreid: A Library for Deep Learning Person Re-Identification in Pytorch](https://mlsanity.com/paper/1910.10093/) (2019)
- [Graph Sampling Based Deep Metric Learning for Generalizable Person Re-Identification](https://mlsanity.com/paper/2104.01546/) (2021)
- [UnrealPerson: An Adaptive Pipeline towards Costless Person Re-identification](https://mlsanity.com/paper/2012.04268/) (2020)
- [TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification](https://mlsanity.com/paper/2105.14432/) (2021)
- [Looking Beyond Appearances: Synthetic Training Data for Deep CNNs in Re-identification](https://mlsanity.com/paper/1701.03153/) (2017)
- [The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching](https://mlsanity.com/paper/1711.08106/) (2017)

## Abstract (arXiv)

Recent studies show that models trained on synthetic datasets are able to achieve better generalizable person re-identification (GPReID) performance than that trained on public real-world datasets. On the other hand, due to the limitations of real-world person ReID datasets, it would also be important and interesting to use large-scale synthetic datasets as test sets to benchmark person ReID algorithms. Yet this raises a critical question: is synthetic dataset reliable for benchmarking generalizable person re-identification? In the literature there is no evidence showing this. To address this, we design a method called Pairwise Ranking Analysis (PRA) to quantitatively measure the ranking similarity and perform the statistical test of identical distributions. Specifically, we employ Kendall rank correlation coefficients to evaluate pairwise similarity values between algorithm rankings on different datasets. Then, a non-parametric two-sample Kolmogorov-Smirnov (KS) test is performed for the judgement of whether algorithm ranking correlations between synthetic and real-world datasets and those only between real-world datasets lie in identical distributions. We conduct comprehensive experiments, with ten representative algorithms, three popular real-world person ReID datasets, and three recently released large-scale synthetic datasets. Through the designed pairwise ranking analysis and comprehensive evaluations, we conclude that a recent large-scale synthetic dataset ClonedPerson can be reliably used to benchmark GPReID, statistically the same as real-world datasets. Therefore, this study guarantees the usage of synthetic datasets for both source training set and target testing set, with completely no privacy concerns from real-world surveillance data. Besides, the study in this paper might also inspire future designs of synthetic datasets.