Data to 7 Oct 2026

AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning

arXiv 2308.07026 ↗First posted 14 Aug 2023cs.CVcited by 101

Not on the map. Why?

Cited by 44 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multimodal contrastive learning aims to train a general-purpose feature extractor, such as CLIP, on vast amounts of raw, unlabeled paired image-text data. This can greatly benefit various complex downstream tasks, including cross-modal image-text retrieval and image classification. Despite its promising prospect, the security issue of cross-modal pre-trained encoder has not been fully explored yet, especially when the pre-trained encoder is publicly available for commercial use. In this work, we propose AdvCLIP, the first attack framework for generating downstream-agnostic adversarial examples based on cross-modal pre-trained encoders. AdvCLIP aims to construct a universal adversarial patch for a set of natural images that can fool all the downstream tasks inheriting the victim cross-modal pre-trained encoder. To address the challenges of heterogeneity between different modalities and unknown downstream tasks, we first build a topological graph structure to capture the relevant positions between target samples and their neighbors. Then, we design a topology-deviation based generative adversarial network to generate a universal adversarial patch. By adding the patch to images, we minimize their embeddings similarity to different modality and perturb the sample distribution in the feature space, achieving unviersal non-targeted attacks. Our results demonstrate the excellent attack performance of AdvCLIP on two types of downstream tasks across eight datasets. We also tailor three popular defenses to mitigate AdvCLIP, highlighting the need for new defense mechanisms to defend cross-modal pre-trained encoders.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

42 of 61 references.

References · 42

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2014Explaining and Harnessing Adversarial Examplescited by 7,287
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
2022Generative Adversarial Networkscited by 5,603
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
2016Universal adversarial perturbationscited by 774
2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709

30 more

2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
2017Adversarial Patchcited by 514
2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
2019Adversarial Training and Robustness for Multiple Perturbationscited by 182
2020A Self-supervised Approach for Adversarial Robustnesscited by 180
2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
2018LaVAN: Localized and Visible Adversarial Noisecited by 112
2018Universal Adversarial Trainingcited by 85
2022Protecting Facial Privacy: Generating Adversarial Identity Masks via Style-robust Makeup Transfercited by 77
2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
2017Fast Feature Fool: A data independent approach to universal adversarial perturbationscited by 65
2016Defensive Distillation is Not Robust to Adversarial Examplescited by 60
2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 50
2021LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrievalcited by 44
2022SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoderscited by 43
2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
2022PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learningcited by 33
2023Downstream-agnostic Adversarial Examplescited by 32
2022BadHash: Invisible Backdoor Attacks against Deep Hashing with Clean Labelcited by 28
2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
2019Design and Interpretation of Universal Adversarial Patches in Face Detectioncited by 22
2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
2022Pre-trained Adversarial Perturbationscited by 16
2017Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Networkcited by 11
2022PointCA: Evaluating the Robustness of 3D Point Cloud Completion Models Against Adversarial Examplescited by 9
2022A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrievalcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 36; 2025: 35; 2026: 21.

101 papers.

Sort:

By area:

2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Modelscited by 43
2023Adversarial Prompt Tuning for Vision-Language Modelscited by 42
2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
2024DarkSAM: Fooling Segment Anything Model to Segment Nothingcited by 32
2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31

38 more

2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
2024Transferable Adversarial Facial Images for Privacy Protectioncited by 26
2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22
2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Modelscited by 21
2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
2024Transferable Adversarial Attacks on SAM and Its Downstream Modelscited by 20
2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
2024NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectorscited by 17
2024A Survey on Safe Multi-Modal Learning Systemcited by 15
2024PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentationcited by 14
2025Test-Time Backdoor Detection for Object Detection Modelscited by 14
2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attackscited by 13
2024Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Featurecited by 13
2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
2025Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2cited by 11
2024Adversarial Backdoor Defense in CLIPcited by 10
2024ECLIPSE: Expunging Clean-label Indiscriminate Poisons via Sparse Diffusion Purificationcited by 9
2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 9
2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
2025Towards Reliable Forgetting: A Survey on Machine Unlearning Verificationcited by 8
2025SegTrans: Transferable Adversarial Examples for Segmentation Modelscited by 8
2025From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledgecited by 8
2024Rethinking Impersonation and Dodging Attacks on Face Recognition Systemscited by 7
2025DarkHash: A Data-Free Backdoor Attack Against Deep Hashingcited by 6
2025Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protectioncited by 5
2026Time Is All It Takes: Spike-Retiming Attacks on Event-Driven Spiking Neural Networkscited by 5
2023SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbationcited by 4
2023MVPatch: More Vivid Patch for Adversarial Camouflaged Attacks on Object Detectors in the Physical Worldcited by 4
2024Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Modelscited by 4
2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Modelscited by 4
2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
2024ProCNS: Progressive Prototype Calibration and Noise Suppression for Weakly-Supervised Medical Image Segmentationcited by 3
2024Adversarial Robustification via Text-to-Image Diffusion Modelscited by 3

50 most cited of 101 All 101 (JSON)