# Structured Semantic Transfer for Multi-Label Recognition with Partial Labels

arXiv: https://arxiv.org/abs/2112.10941 · HTML: https://mlsanity.com/paper/2112.10941/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Learning a Deep ConvNet for Multi-label Classification with Partial Labels](https://mlsanity.com/paper/1902.09720/) (2019)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Multi-label Image Recognition by Recurrently Discovering Attentional Regions](https://mlsanity.com/paper/1711.02816/) (2017)
- [Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognition](https://mlsanity.com/paper/2012.02994/) (2020)
- [Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition](https://mlsanity.com/paper/2009.09450/) (2020)
- [CNN: Single-label to Multi-label](https://mlsanity.com/paper/1406.5726/) (2014)
- [Large-scale Multi-label Learning with Missing Labels](https://mlsanity.com/paper/1307.5101/) (2013)
- [Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning](https://mlsanity.com/paper/2008.00923/) (2020)
- [Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition](https://mlsanity.com/paper/1712.07465/) (2017)
- [Exploit Bounding Box Annotations for Multi-label Object Recognition](https://mlsanity.com/paper/1504.05843/) (2015)
- [Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition](https://mlsanity.com/paper/1807.00505/) (2018)

## Built on by (most cited first, 30 of 41)

- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [Texts as Images in Prompt Tuning for Multi-Label Image Recognition](https://mlsanity.com/paper/2211.12739/) (2022)
- [CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classification](https://mlsanity.com/paper/2307.16634/) (2023)
- [Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2203.02172/) (2022)
- [Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels](https://mlsanity.com/paper/2303.13223/) (2023)
- [DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2308.01890/) (2023)
- [Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels](https://mlsanity.com/paper/2205.11131/) (2022)
- [Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration](https://mlsanity.com/paper/2407.06844/) (2024)
- [Bridging the Gap between Model Explanations in Partially Annotated Multi-label Classification](https://mlsanity.com/paper/2304.01804/) (2023)
- [Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation](https://mlsanity.com/paper/2309.13237/) (2023)
- [Text-Region Matching for Multi-Label Image Recognition with Missing Labels](https://mlsanity.com/paper/2407.18520/) (2024)
- [Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning](https://mlsanity.com/paper/2403.01209/) (2024)
- [Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training](https://mlsanity.com/paper/2404.06287/) (2024)
- [Semantic-Aware Graph Matching Mechanism for Multi-Label Image Recognition](https://mlsanity.com/paper/2304.11275/) (2023)
- [Positive Label Is All You Need for Multi-Label Classification](https://mlsanity.com/paper/2306.16016/) (2023)
- [Semantic Representation and Dependency Learning for Multi-Label Image Recognition](https://mlsanity.com/paper/2204.03795/) (2022)
- [Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2205.13092/) (2022)
- [Semantic Contrastive Bootstrapping for Single-positive Multi-label Recognition](https://mlsanity.com/paper/2307.07680/) (2023)
- [Balancing Efficiency vs. Effectiveness and Providing Missing Label Robustness in Multi-Label Stream Classification](https://mlsanity.com/paper/2310.00665/) (2023)
- [SpliceMix: A Cross-scale and Semantic Blending Augmentation Strategy for Multi-label Image Classification](https://mlsanity.com/paper/2311.15200/) (2023)
- [Text as Image: Learning Transferable Adapter for Multi-Label Classification](https://mlsanity.com/paper/2312.04160/) (2023)
- [Classifier-guided CLIP Distillation for Unsupervised Multi-label Classification](https://mlsanity.com/paper/2503.16873/) (2025)
- [Category-wise Fine-Tuning: Resisting Incorrect Pseudo-Labels in Multi-Label Image Classification with Partial Labels](https://mlsanity.com/paper/2401.16991/) (2024)
- [Labeled-to-Unlabeled Distribution Alignment for Partially-Supervised Multi-Organ Medical Image Segmentation](https://mlsanity.com/paper/2409.03228/) (2024)
- [Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations](https://mlsanity.com/paper/2409.08381/) (2024)
- [Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels](https://mlsanity.com/paper/2412.10843/) (2024)
- [Multi-Grained Compositional Visual Clue Learning for Image Intent Recognition](https://mlsanity.com/paper/2504.18201/) (2025)
- [Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning](https://mlsanity.com/paper/2506.10575/) (2025)
- [Exploring Partial Multi-Label Learning via Integrating Semantic Co-occurrence Knowledge](https://mlsanity.com/paper/2507.05992/) (2025)
- [PSScreen: Partially Supervised Multiple Retinal Disease Screening](https://mlsanity.com/paper/2508.10549/) (2025)

## Abstract (arXiv)

Multi-label image recognition is a fundamental yet practical task because real-world images inherently possess multiple semantic labels. However, it is difficult to collect large-scale multi-label annotations due to the complexity of both the input images and output label spaces. To reduce the annotation cost, we propose a structured semantic transfer (SST) framework that enables training multi-label recognition models with partial labels, i.e., merely some labels are known while other labels are missing (also called unknown labels) per image. The framework consists of two complementary transfer modules that explore within-image and cross-image semantic correlations to transfer knowledge of known labels to generate pseudo labels for unknown labels. Specifically, an intra-image semantic transfer module learns image-specific label co-occurrence matrix and maps the known labels to complement unknown labels based on this matrix. Meanwhile, a cross-image transfer module learns category-specific feature similarities and helps complement unknown labels with high similarities. Finally, both known and generated labels are used to train the multi-label recognition models. Extensive experiments on the Microsoft COCO, Visual Genome and Pascal VOC datasets show that the proposed SST framework obtains superior performance over current state-of-the-art algorithms. Codes are available at https://github.com/HCPLab-SYSU/HCP-MLR-PL.