mlsanityData to 7 Oct 2026

Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models

arXiv 2408.13979 ↗First posted 26 Aug 2024cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: ``Do we need to normalize the soft prompts in VLMs?'' To fill this research gap, we first uncover a phenomenon, called the Low-Norm Effect by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named Normalizing the soft-prompt vectors of vision-language models (Nemesis) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning. The code is available at texttthrefhttps://github.com/ShyFoo/Nemesishttps://github.com/ShyFoo/Nemesis.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 40 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

29 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Unified Vision and Language Prompt Learning13 Oct
  2. 2022Prompt Distribution Learning6 May
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Contrastive Vision-Language Pre-training with Limited Resources17 Dec
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
11 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  6. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  7. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  8. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  9. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  10. 2023On the Role of Attention in Prompt-tuningcited by 49
  11. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  2. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
  2. 2024Generalizable Prompt Tuning for Vision-Language Models
  3. 2025Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise
  4. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models
  5. 2025GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
  6. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
  7. 2024Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
  8. 2026Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models