mlsanityData to 14 Oct 2026

Vision-Language Model IP Protection via Prompt-based Learning

arXiv 2503.02393 ↗First posted 4 Mar 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) like CLIP (Contrastive Language-Image Pre-Training) have seen remarkable success in visual recognition, highlighting the increasing need to safeguard the intellectual property (IP) of well-trained models. Effective IP protection extends beyond ensuring authorized usage; it also necessitates restricting model deployment to authorized data domains, particularly when the model is fine-tuned for specific target domains. However, current IP protection methods often rely solely on the visual backbone, which may lack sufficient semantic richness. To bridge this gap, we introduce IP-CLIP, a lightweight IP protection strategy tailored to CLIP, employing a prompt-based learning approach. By leveraging the frozen visual backbone of CLIP, we extract both image style and content information, incorporating them into the learning of IP prompt. This strategy acts as a robust barrier, effectively preventing the unauthorized transfer of features from authorized domains to unauthorized ones. Additionally, we propose a style-enhancement branch that constructs feature banks for both authorized and unauthorized domains. This branch integrates self-enhanced and cross-domain features, further strengthening IP-CLIP's capability to block features from unauthorized domains. Finally, we present new three metrics designed to better balance the performance degradation of authorized and unauthorized domains. Comprehensive experiments in various scenarios demonstrate its promising potential for application in IP protection tasks for VLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

28 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP26 Nov

Other references · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Segment Anythingcited by 8,234
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
15 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  2. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  5. 2020Domain Adaptive Ensemble Learningcited by 156
  6. 2017Adversarial Frontier Stitching for Remote Neural Network Watermarkingcited by 118
  7. 2022Fingerprinting Deep Neural Networks Globally via Universal Adversarial Perturbationscited by 61
  8. 2021Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorizationcited by 52
  9. 2020Intellectual Property Protection for Deep Learning Models: Taxonomy, Methods, Attacks, and Evaluationscited by 31
  10. 2022Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networkscited by 21
  11. 2023Uncertainty-inspired Open Set Learning for Retinal Anomaly Identificationcited by 18
  12. 2023Model Barrier: A Compact Un-Transferable Isolation Domain for Model Intellectual Property Protectioncited by 14
  13. 2024Say No to Freeloader: Protecting Intellectual Property of Your Deep Modelcited by 7
  14. 2022Unsupervised Non-transferable Text Classificationcited by 6
  15. 2024Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for VPaaS via Watermarkcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothingcited by 3
  2. 2026SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Modelscited by 2
  3. 2026Authorize-on-Demand: Dynamic Authorization with Legality-Aware Intellectual Property Protection for VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images
  2. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulated
  3. 2025Can Large Vision-Language Models Detect Images Copyright Infringement from GenAI?
  4. 2025SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking
  5. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  6. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks
  7. 2025VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
  8. 2026Text is All You Need for Vision-Language Model Jailbreaking