# Self-Supervised Pre-Training for Transformer-Based Person Re-Identification

arXiv: https://arxiv.org/abs/2111.12084 · HTML: https://mlsanity.com/paper/2111.12084/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 23 Nov 2021.

## Builds on (most cited first, 30 of 39)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Bag of Tricks and A Strong Baseline for Deep Person Re-identification](https://mlsanity.com/paper/1903.07071/) (2019)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Learning Discriminative Features with Multiple Granularities for Person Re-Identification](https://mlsanity.com/paper/1804.01438/) (2018)
- [Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification](https://mlsanity.com/paper/2001.01526/) (2020)
- [Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID](https://mlsanity.com/paper/2006.02713/) (2020)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [A Strong Baseline and Batch Normalization Neck for Deep Person Re-identification](https://mlsanity.com/paper/1906.08332/) (2019)
- [ABD-Net: Attentive but Diverse Person Re-Identification](https://mlsanity.com/paper/1908.01114/) (2019)
- [Self-Supervised Learning with Swin Transformers](https://mlsanity.com/paper/2105.04553/) (2021)
- [Diverse Part Discovery: Occluded Person Re-identification with Part-Aware Transformer](https://mlsanity.com/paper/2106.04095/) (2021)
- [Unsupervised Pre-training for Person Re-identification](https://mlsanity.com/paper/2012.03753/) (2020)
- [Cluster Contrast for Unsupervised Person Re-Identification](https://mlsanity.com/paper/2103.11568/) (2021)
- [Hierarchical Clustering with Hard-batch Triplet Loss for Person Re-identification](https://mlsanity.com/paper/1910.12278/) (2019)
- [Joint Disentangling and Adaptation for Cross-Domain Person Re-Identification](https://mlsanity.com/paper/2007.10315/) (2020)
- [Unsupervised Person Re-identification by Soft Multilabel Learning](https://mlsanity.com/paper/1903.06325/) (2019)

## Built on by (most cited first, 30 of 36)

- [Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks](https://mlsanity.com/paper/2303.17602/) (2023)
- [Part-Aware Transformer for Generalizable Person Re-identification](https://mlsanity.com/paper/2308.03322/) (2023)
- [HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception](https://mlsanity.com/paper/2310.20695/) (2023)
- [SoccerNet 2022 Challenges Results](https://mlsanity.com/paper/2210.02365/) (2022)
- [Transformer for Object Re-Identification: A Survey](https://mlsanity.com/paper/2401.06960/) (2024)
- [The 7th AI City Challenge](https://mlsanity.com/paper/2304.07500/) (2023)
- [Unified Pre-training with Pseudo Texts for Text-To-Image Person Re-identification](https://mlsanity.com/paper/2309.01420/) (2023)
- [All in One Framework for Multimodal Re-identification in the Wild](https://mlsanity.com/paper/2405.04741/) (2024)
- [DC-Former: Diverse and Compact Transformer for Person Re-Identification](https://mlsanity.com/paper/2302.14335/) (2023)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)
- [Triplet Contrastive Representation Learning for Unsupervised Vehicle Re-identification](https://mlsanity.com/paper/2301.09498/) (2023)
- [A Versatile Framework for Multi-scene Person Re-identification](https://mlsanity.com/paper/2403.11121/) (2024)
- [PersonViT: Large-scale Self-supervised Vision Transformer for Person Re-Identification](https://mlsanity.com/paper/2408.05398/) (2024)
- [PASS: Part-Aware Self-Supervised Pre-Training for Person Re-Identification](https://mlsanity.com/paper/2203.03931/) (2022)
- [Cross-video Identity Correlating for Person Re-identification Pre-training](https://mlsanity.com/paper/2409.18569/) (2024)
- [Cross-modal Ship Re-Identification via Optical and SAR Imagery: A Novel Dataset and Method](https://mlsanity.com/paper/2506.22027/) (2025)
- [Transformer Based Multi-Grained Features for Unsupervised Person Re-Identification](https://mlsanity.com/paper/2211.12280/) (2022)
- [PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoders](https://mlsanity.com/paper/2311.04496/) (2023)
- [Towards All-in-One Medical Image Re-Identification](https://mlsanity.com/paper/2503.08173/) (2025)
- [One-Shot Knowledge Transfer for Scalable Person Re-Identification](https://mlsanity.com/paper/2511.06016/) (2025)
- [Learning Transferable Pedestrian Representation from Multimodal Information Supervision](https://mlsanity.com/paper/2304.05554/) (2023)
- [MVIP -- A Dataset and Methods for Application Oriented Multi-View and Multi-Modal Industrial Part Recognition](https://mlsanity.com/paper/2502.15448/) (2025)
- [DenoiseRep: Denoising Model for Representation Learning](https://mlsanity.com/paper/2406.08773/) (2024)
- [Generalizable Re-Identification from Videos with Cycle Association](https://mlsanity.com/paper/2211.03663/) (2022)
- [Exploring the Camera Bias of Person Re-identification](https://mlsanity.com/paper/2502.10195/) (2025)
- [Not All Starting Points Are Equal: Pre-trained Priors and Their Outsized Impact on Person Identification](https://mlsanity.com/paper/2507.17640/) (2025)
- [PGDS: Pose-Guidance Deep Supervision for Mitigating Clothes-Changing in Person Re-Identification](https://mlsanity.com/paper/2312.05634/) (2023)
- [Mutual Distillation Learning For Person Re-Identification](https://mlsanity.com/paper/2401.06430/) (2024)
- [Contrastive Multiple Instance Learning for Weakly Supervised Person ReID](https://mlsanity.com/paper/2402.07685/) (2024)
- [Privacy-Preserving Adaptive Re-Identification without Image Transfer](https://mlsanity.com/paper/2407.12589/) (2024)

## Abstract (arXiv)

Transformer-based supervised pre-training achieves great performance in person re-identification (ReID). However, due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset (e.g. ImageNet-21K) to boost the performance because of the strong data fitting ability of the transformer. To address this challenge, this work targets to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure, respectively. We first investigate self-supervised learning (SSL) methods with Vision Transformer (ViT) pretrained on unlabelled person images (the LUPerson dataset), and empirically find it significantly surpasses ImageNet supervised pre-training models on ReID tasks. To further reduce the domain gap and accelerate the pre-training, the Catastrophic Forgetting Score (CFS) is proposed to evaluate the gap between pre-training and fine-tuning data. Based on CFS, a subset is selected via sampling relevant data close to the down-stream ReID data and filtering irrelevant data from the pre-training dataset. For the model structure, a ReID-specific module named IBN-based convolution stem (ICS) is proposed to bridge the domain gap by learning more invariant features. Extensive experiments have been conducted to fine-tune the pre-training models under supervised learning, unsupervised domain adaptation (UDA), and unsupervised learning (USL) settings. We successfully downscale the LUPerson dataset to 50% with no performance degradation. Finally, we achieve state-of-the-art performance on Market-1501 and MSMT17. For example, our ViT-S/16 achieves 91.3%/89.9%/89.6% mAP accuracy on Market1501 for supervised/UDA/USL ReID. Codes and models will be released to https://github.com/michuanhaohao/TransReID-SSL.