# LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition

arXiv: https://arxiv.org/abs/2305.04536 · HTML: https://mlsanity.com/paper/2305.04536/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 61 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.05830/) (2015)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [ResLT: Residual Learning for Long-tailed Recognition](https://mlsanity.com/paper/2101.10633/) (2021)
- [Distributional Robustness Loss for Long-tail Learning](https://mlsanity.com/paper/2104.03066/) (2021)
- [Multi-label Image Recognition by Recurrently Discovering Attentional Regions](https://mlsanity.com/paper/1711.02816/) (2017)
- [Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognition](https://mlsanity.com/paper/2012.02994/) (2020)

## Built on by (most cited first, 15 of 15)

- [CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models](https://mlsanity.com/paper/2406.06007/) (2024)
- [RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models](https://mlsanity.com/paper/2407.05131/) (2024)
- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding](https://mlsanity.com/paper/2311.14064/) (2023)
- [MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization](https://mlsanity.com/paper/2412.06141/) (2024)
- [NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding](https://mlsanity.com/paper/2310.13347/) (2023)
- [Generalizing to Unseen Domains in Diabetic Retinopathy with Disentangled Representations](https://mlsanity.com/paper/2406.06384/) (2024)
- [Long-tailed Recognition with Model Rebalancing](https://mlsanity.com/paper/2510.08177/) (2025)
- [Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classification](https://mlsanity.com/paper/2408.08125/) (2024)
- [LIFT+: Lightweight Fine-Tuning for Long-Tail Learning](https://mlsanity.com/paper/2504.13282/) (2025)
- [Diffusion Model Driven Test-Time Image Adaptation for Robust Skin Lesion Classification](https://mlsanity.com/paper/2405.11289/) (2024)
- [Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classification](https://mlsanity.com/paper/2509.17747/) (2025)
- [Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition](https://mlsanity.com/paper/2511.20641/) (2025)
- [CUE: Concept-Aware Multi-Label Expansion to Mitigate Concept Confusion in Long-Tailed Learning](https://mlsanity.com/paper/2605.01309/) (2026)
- [Rethinking Multi-Label Image Classification With Deep Learning: Taxonomy, Challenge, and Outlook](https://mlsanity.com/paper/2607.00839/) (2026)

## Abstract (arXiv)

Long-tailed multi-label visual recognition (LTML) task is a highly challenging task due to the label co-occurrence and imbalanced data distribution. In this work, we propose a unified framework for LTML, namely prompt tuning with class-specific embedding loss (LMPT), capturing the semantic feature interactions between categories by combining text and image modality data and improving the performance synchronously on both head and tail classes. Specifically, LMPT introduces the embedding loss function with class-aware soft margin and re-weighting to learn class-specific contexts with the benefit of textual descriptions (captions), which could help establish semantic relationships between classes, especially between the head and tail classes. Furthermore, taking into account the class imbalance, the distribution-balanced loss is adopted as the classification loss function to further improve the performance on the tail classes without compromising head classes. Extensive experiments are conducted on VOC-LT and COCO-LT datasets, which demonstrates that our method significantly surpasses the previous state-of-the-art methods and zero-shot CLIP in LTML. Our codes are fully public at https://github.com/richard-peng-xia/LMPT.