mlsanityData to 7 Oct 2026

GalLoP: Learning Global and Local Prompts for Vision-Language Models

arXiv 2407.01400 ↗First posted 1 Jul 2024cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs), e.g. CLIP, for few-shot image classification. Despite their success, most prompt learning methods trade-off between classification accuracy and robustness, e.g. in domain generalization or out-of-distribution (OOD) detection. In this work, we introduce Global-Local Prompts (GalLoP), a new prompt learning method that learns multiple diverse prompts leveraging both global and local visual features. The training of the local prompts relies on local features with an enhanced vision-text alignment. To focus only on pertinent features, this local alignment is coupled with a sparsity strategy in the selection of the local features. We enforce diversity on the set of prompts using a new ``prompt dropout'' technique and a multiscale strategy on the local prompts. GalLoP outperforms previous prompt learning methods on accuracy on eleven datasets in different few shots settings and with various backbones. Furthermore, GalLoP shows strong robustness performances in both domain generalization and OOD detection, even outperforming dedicated OOD detection methods. Code and instructions to reproduce our results: https://github.com/MarcLafon/gallop.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 52 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

38 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023ECO: Ensembling Context Optimization for Vision-Language Models26 Jul
  2. 2022Learning to Decompose Visual Features with Latent Textual Prompts9 Oct
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Prompt Distribution Learning6 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
19 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  6. 2021Extract Free Dense Labels from CLIPcited by 580
  7. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  8. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  9. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  10. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  11. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  12. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  13. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  14. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  15. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  16. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  17. 2023Domain Aligned CLIP for Few-shot Classificationcited by 16
  18. 2023Learning to Name Classes for Vision and Language Modelscited by 12
  19. 2023Hybrid Energy Based Model in the Feature Space for Out-of-Distribution Detectioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 8.

29 papers.

Sort:

By area:

  1. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  2. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  3. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  4. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  5. 2025Recover and Match: Open-Vocabulary Multi-Label Recognition through Knowledge-Constrained Optimal Transportcited by 6
  6. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  7. 2025Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Surveycited by 5
  8. 2025VTD-CLIP: Video-to-Text Discretization via Prompting CLIPcited by 2
  9. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Modelscited by 2
  10. 2025Simulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalizationcited by 2
  11. 2025Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detectioncited by 2
  12. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulationcited by 2
17 more
  1. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  2. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  3. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  4. 2026ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Researchcited by 1
  5. 2025PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Modelscited by 0
  6. 2025Weighted Multi-Prompt Learning with Description-free Large Language Model Distillationcited by 0
  7. 2025Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learningcited by 0
  8. 2025Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representationscited by 0
  9. 2025Attn-Adapter: Attention Is All You Need for Online Few-shot Learner of Vision-Language Modelcited by 0
  10. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classificationcited by 0
  11. 2026Local-Global Prompt Learning via Sparse Optimal Transportcited by 0
  12. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0
  13. 2026Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statisticscited by 0
  14. 2026Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Modelcited by 0
  15. 2026Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learningcited by 0
  16. 2026NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Modelscited by 0
  17. 2026TecoPrompt: Temporal-Conservative Prompt Learning for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LaViP:Language-Grounded Visual Prompts
  2. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  3. 2023DPL: Decoupled Prompt Learning for Vision-Language Models
  4. 2024Revisiting Prompt Pretraining of Vision-Language Models
  5. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection
  6. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
  7. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
  8. 2023Towards Robust Prompts on Vision-Language Models