# Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels

arXiv: https://arxiv.org/abs/2205.11131 · HTML: https://mlsanity.com/paper/2205.11131/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Learning a Deep ConvNet for Multi-label Classification with Partial Labels](https://mlsanity.com/paper/1902.09720/) (2019)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Unsupervised Person Re-identification via Multi-label Classification](https://mlsanity.com/paper/2004.09228/) (2020)
- [Multi-label Image Recognition by Recurrently Discovering Attentional Regions](https://mlsanity.com/paper/1711.02816/) (2017)
- [Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognition](https://mlsanity.com/paper/2012.02994/) (2020)
- [Unsupervised Person Re-identification: Clustering and Fine-tuning](https://mlsanity.com/paper/1705.10444/) (2017)
- [Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition](https://mlsanity.com/paper/2009.09450/) (2020)
- [CNN: Single-label to Multi-label](https://mlsanity.com/paper/1406.5726/) (2014)
- [Large-scale Multi-label Learning with Missing Labels](https://mlsanity.com/paper/1307.5101/) (2013)
- [Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning](https://mlsanity.com/paper/2008.00923/) (2020)
- [Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition](https://mlsanity.com/paper/1712.07465/) (2017)
- [Structured Semantic Transfer for Multi-Label Recognition with Partial Labels](https://mlsanity.com/paper/2112.10941/) (2021)
- [Exploit Bounding Box Annotations for Multi-label Object Recognition](https://mlsanity.com/paper/1504.05843/) (2015)
- [Contextualized Spatial-Temporal Network for Taxi Origin-Destination Demand Prediction](https://mlsanity.com/paper/1905.06335/) (2019)
- [Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2203.02172/) (2022)
- [Intra-Camera Supervised Person Re-Identification](https://mlsanity.com/paper/2002.05046/) (2020)
- [Instance-weighted Central Similarity for Multi-label Image Retrieval](https://mlsanity.com/paper/2108.05274/) (2021)

## Built on by (most cited first, 19 of 19)

- [StyleAdapter: A Unified Stylized Image Generation Model](https://mlsanity.com/paper/2309.01770/) (2023)
- [Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration](https://mlsanity.com/paper/2407.06844/) (2024)
- [Contrastive Decoupled Representation Learning and Regularization for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2504.05672/) (2025)
- [SQLNet: Scale-Modulated Query and Localization Network for Few-Shot Class-Agnostic Counting](https://mlsanity.com/paper/2311.10011/) (2023)
- [Globally Correlation-Aware Hard Negative Generation](https://mlsanity.com/paper/2411.13145/) (2024)
- [Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation](https://mlsanity.com/paper/2504.05746/) (2025)
- [Neural Scene Designer: Self-Styled Semantic Image Manipulation](https://mlsanity.com/paper/2509.01405/) (2025)
- [Category-wise Fine-Tuning: Resisting Incorrect Pseudo-Labels in Multi-Label Image Classification with Partial Labels](https://mlsanity.com/paper/2401.16991/) (2024)
- [Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels](https://mlsanity.com/paper/2412.10843/) (2024)
- [Exploring Partial Multi-Label Learning via Integrating Semantic Co-occurrence Knowledge](https://mlsanity.com/paper/2507.05992/) (2025)
- [PSScreen: Partially Supervised Multiple Retinal Disease Screening](https://mlsanity.com/paper/2508.10549/) (2025)
- [Free Performance Gain from Mixing Multiple Partially Labeled Samples in Multi-label Image Classification](https://mlsanity.com/paper/2405.15860/) (2024)
- [Learning Semantic-Aware Threshold for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2507.23263/) (2025)
- [PSScreen V2: Partially Supervised Multiple Retinal Disease Screening](https://mlsanity.com/paper/2510.22589/) (2025)
- [Investigating ECG Diagnosis with Ambiguous Labels using Partial Label Learning](https://mlsanity.com/paper/2512.11095/) (2025)
- [Neural Clothing Tryer: Customized Virtual Try-On via Semantic Enhancement and Controlling Diffusion Model](https://mlsanity.com/paper/2601.22838/) (2026)
- [D2ACE: Multi-Label Batch Selection Guided by Dual Dynamics and Adaptive Correlation Enhancement](https://mlsanity.com/paper/2605.09400/) (2026)
- [Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning](https://mlsanity.com/paper/2607.20874/) (2026)
- [Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label](https://mlsanity.com/paper/2608.22313/) (2026)

## Abstract (arXiv)

Multi-label image recognition with partial labels (MLR-PL), in which some labels are known while others are unknown for each image, may greatly reduce the cost of annotation and thus facilitate large-scale MLR. We find that strong semantic correlations exist within each image and across different images, and these correlations can help transfer the knowledge possessed by the known labels to retrieve the unknown labels and thus improve the performance of the MLR-PL task (see Figure 1). In this work, we propose a novel heterogeneous semantic transfer (HST) framework that consists of two complementary transfer modules that explore both within-image and cross-image semantic correlations to transfer the knowledge possessed by known labels to generate pseudo labels for the unknown labels. Specifically, an intra-image semantic transfer (IST) module learns an image-specific label co-occurrence matrix for each image and maps the known labels to complement the unknown labels based on these matrices. Additionally, a cross-image transfer (CST) module learns category-specific feature-prototype similarities and then helps complement the unknown labels that have high degrees of similarity with the corresponding prototypes. Finally, both the known and generated pseudo labels are used to train MLR models. Extensive experiments conducted on the Microsoft COCO, Visual Genome, and Pascal VOC 2007 datasets show that the proposed HST framework achieves superior performance to that of current state-of-the-art algorithms. Specifically, it obtains mean average precision (mAP) improvements of 1.4%, 3.3%, and 0.4% on the three datasets over the results of the best-performing previously developed algorithm.