# LAMM: Label Alignment for Multi-Modal Prompt Learning

arXiv: https://arxiv.org/abs/2312.08212 · HTML: https://mlsanity.com/paper/2312.08212/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)

## Built on by (most cited first, 15 of 15)

- [Enhance Vision-Language Alignment with Noise](https://mlsanity.com/paper/2412.10817/) (2024)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detection](https://mlsanity.com/paper/2506.17633/) (2025)
- [Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytelling](https://mlsanity.com/paper/2408.03695/) (2024)
- [Learning Multi-axis Representation in Frequency Domain for Medical Image Segmentation](https://mlsanity.com/paper/2312.17030/) (2023)
- [FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion](https://mlsanity.com/paper/2410.12564/) (2024)
- [iDAT: inverse Distillation Adapter-Tuning](https://mlsanity.com/paper/2403.15750/) (2024)
- [Data-Efficient CLIP-Powered Dual-Branch Networks for Source-Free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2410.15811/) (2024)
- [Diversity Covariance-Aware Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2503.01531/) (2025)
- [Understanding Robustness of Parameter-Efficient Tuning for Image Classification](https://mlsanity.com/paper/2410.09845/) (2024)
- [FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation](https://mlsanity.com/paper/2505.18053/) (2025)
- [UMCL: Unimodal-generated Multimodal Contrastive Learning for Cross-compression-rate Deepfake Detection](https://mlsanity.com/paper/2511.18983/) (2025)
- [How to Understand Named Entities: Using Common Sense for News Captioning](https://mlsanity.com/paper/2403.06520/) (2024)
- [Integrated Structural Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.05677/) (2025)
- [Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning](https://mlsanity.com/paper/2605.16806/) (2026)

## Abstract (arXiv)

With the success of pre-trained visual-language (VL) models such as CLIP in visual representation tasks, transferring pre-trained models to downstream tasks has become a crucial paradigm. Recently, the prompt tuning paradigm, which draws inspiration from natural language processing (NLP), has made significant progress in VL field. However, preceding methods mainly focus on constructing prompt templates for text and visual inputs, neglecting the gap in class label representations between the VL models and downstream tasks. To address this challenge, we introduce an innovative label alignment method named LAMM, which can dynamically adjust the category embeddings of downstream datasets through end-to-end training. Moreover, to achieve a more appropriate label distribution, we propose a hierarchical loss, encompassing the alignment of the parameter space, feature space, and logits space. We conduct experiments on 11 downstream vision datasets and demonstrate that our method significantly improves the performance of existing multi-modal prompt learning models in few-shot scenarios, exhibiting an average accuracy improvement of 2.31(%) compared to the state-of-the-art methods on 16 shots. Moreover, our methodology exhibits the preeminence in continual learning compared to other prompt tuning methods. Importantly, our method is synergistic with existing prompt tuning methods and can boost the performance on top of them. Our code and dataset will be publicly available at https://github.com/gaojingsheng/LAMM.