# Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks

arXiv: https://arxiv.org/abs/2303.17602 · HTML: https://mlsanity.com/paper/2303.17602/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 111 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 68)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Deep Learning for Person Re-identification: A Survey and Outlook](https://mlsanity.com/paper/2001.04193/) (2020)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Joint Detection and Identification Feature Learning for Person Search](https://mlsanity.com/paper/1604.01850/) (2016)
- [Self-Correction for Human Parsing](https://mlsanity.com/paper/1910.09777/) (2019)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Person Re-identification in the Wild](https://mlsanity.com/paper/1604.02531/) (2016)
- [HRFormer: High-Resolution Transformer for Dense Prediction](https://mlsanity.com/paper/2110.09408/) (2021)

## Built on by (most cited first, 30 of 96)

- [Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark](https://mlsanity.com/paper/2306.02898/) (2023)
- [PLIP: Language-Image Pre-training for Person Representation Learning](https://mlsanity.com/paper/2305.08386/) (2023)
- [Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions](https://mlsanity.com/paper/2306.07520/) (2023)
- [HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception](https://mlsanity.com/paper/2310.20695/) (2023)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [Transformer for Object Re-Identification: A Survey](https://mlsanity.com/paper/2401.06960/) (2024)
- [Referring to Any Person](https://mlsanity.com/paper/2503.08507/) (2025)
- [Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation](https://mlsanity.com/paper/2403.05239/) (2024)
- [From Poses to Identity: Training-Free Person Re-Identification via Feature Centralization](https://mlsanity.com/paper/2503.00938/) (2025)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)
- [Keypoint Promptable Re-Identification](https://mlsanity.com/paper/2407.18112/) (2024)
- [SapiensID: Foundation for Human Recognition](https://mlsanity.com/paper/2504.04708/) (2025)
- [Human-Centric Foundation Models: Perception, Generation and Agentic Modeling](https://mlsanity.com/paper/2502.08556/) (2025)
- [Synthesizing Efficient Data with Diffusion Models for Person Re-Identification Pre-Training](https://mlsanity.com/paper/2406.06045/) (2024)
- [CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering](https://mlsanity.com/paper/2311.15510/) (2023)
- [You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perception](https://mlsanity.com/paper/2312.05525/) (2023)
- [PersonViT: Large-scale Self-supervised Vision Transformer for Person Re-Identification](https://mlsanity.com/paper/2408.05398/) (2024)
- [MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos](https://mlsanity.com/paper/2412.00692/) (2024)
- [Human as Points: Explicit Point-based 3D Human Reconstruction from Single-view RGB Images](https://mlsanity.com/paper/2311.02892/) (2023)
- [Explore Human Parsing Modality for Action Recognition](https://mlsanity.com/paper/2401.02138/) (2024)
- [PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation](https://mlsanity.com/paper/2409.06535/) (2024)
- [Cross-video Identity Correlating for Person Re-identification Pre-training](https://mlsanity.com/paper/2409.18569/) (2024)
- [ReMix: Training Generalized Person Re-identification on a Mixture of Data](https://mlsanity.com/paper/2410.21938/) (2024)
- [ICPL-ReID: Identity-Conditional Prompt Learning for Multi-Spectral Object Re-Identification](https://mlsanity.com/paper/2505.17821/) (2025)
- [Cross-modal Ship Re-Identification via Optical and SAR Imagery: A Novel Dataset and Method](https://mlsanity.com/paper/2506.22027/) (2025)
- [GraFT: Gradual Fusion Transformer for Multimodal Re-Identification](https://mlsanity.com/paper/2310.16856/) (2023)
- [Enhancing person re-identification via Uncertainty Feature Fusion Method and Auto-weighted Measure Combination](https://mlsanity.com/paper/2405.01101/) (2024)
- [Pose-dIVE: Pose-Diversified Augmentation with Diffusion Model for Person Re-Identification](https://mlsanity.com/paper/2406.16042/) (2024)
- [A Culturally-Aware Benchmark for Person Re-Identification in Modest Attire](https://mlsanity.com/paper/2412.18874/) (2024)
- [Towards All-in-One Medical Image Re-Identification](https://mlsanity.com/paper/2503.08173/) (2025)

## Abstract (arXiv)

Human-centric visual tasks have attracted increasing research attention due to their widespread applications. In this paper, we aim to learn a general human representation from massive unlabeled human images which can benefit downstream human-centric tasks to the maximum extent. We call this method SOLIDER, a Semantic cOntrollable seLf-supervIseD lEaRning framework. Unlike the existing self-supervised learning methods, prior knowledge from human images is utilized in SOLIDER to build pseudo semantic labels and import more semantic information into the learned representation. Meanwhile, we note that different downstream tasks always require different ratios of semantic information and appearance information. For example, human parsing requires more semantic information, while person re-identification needs more appearance information for identification purpose. So a single learned representation cannot fit for all requirements. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After the model is trained, users can send values to the controller to produce representations with different ratios of semantic information, which can fit different needs of downstream tasks. Finally, SOLIDER is verified on six downstream human-centric visual tasks. It outperforms state of the arts and builds new baselines for these tasks. The code is released in https://github.com/tinyvision/SOLIDER.