mlsanityData to 14 Oct 2026

Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

arXiv 2502.18290 ↗First posted 25 Feb 2025cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Self-supervised learning (SSL) vision encoders learn high-quality image representations and thus have become a vital part of developing vision modality of large vision language models (LVLMs). Due to the high cost of training such encoders, pre-trained encoders are widely shared and deployed into many LVLMs, which are security-critical or bear societal significance. Under this practical scenario, we reveal a new backdoor threat that significant visual hallucinations can be induced into these LVLMs by merely compromising vision encoders. Because of the sharing and reuse of these encoders, many downstream LVLMs may inherit backdoor behaviors from encoders, leading to widespread backdoors. In this work, we propose BadVision, the first method to exploit this vulnerability in SSL vision encoders for LVLMs with novel trigger optimization and backdoor learning techniques. We evaluate BadVision on two types of SSL encoders and LVLMs across eight benchmarks. We show that BadVision effectively drives the LVLMs to attacker-chosen hallucination with over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining the stealthiness. SoTA backdoor detection methods cannot detect our attack effectively.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 60 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

46 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Backdooring Vision-Language Models with Out-Of-Distribution Data2 Oct
  2. 2024TrojVLM: Backdoor Attack Against Vision Language Models28 Sep
  3. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  4. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  5. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  6. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
27 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  7. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  8. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  9. 2020Debiased Contrastive Learningcited by 356
  10. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  11. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  12. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  13. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  14. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  15. 2021Poisoning and Backdooring Contrastive Learningcited by 151
  16. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  17. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  18. 2020Captioning Images Taken by People Who Are Blindcited by 110
  19. 2021Backdoor Attacks on Self-Supervised Learningcited by 90
  20. 2022The "Beatrix'' Resurrections: Robust Backdoor Detection via Gram Matricescited by 63
  21. 2023Detecting Backdoors in Pre-trained Encoderscited by 55
  22. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  23. 2019Self-Supervised Learning For Few-Shot Image Classificationcited by 44
  24. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 33
  25. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 31
  26. 2022CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learningcited by 29
  27. 2023GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulationcited by 14
  2. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulatedcited by 12
  3. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12
  4. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 5
  5. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  6. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  7. 2026Visual Memory Injection Attacks for Multi-Turn Conversationscited by 2
  8. 2026BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encodercited by 1
  9. 2026Position: 3D Gaussian Splatting Watermarking Should Be Scenario-Driven and Threat-Model Explicitcited by 0
  10. 2026Phantasia: Context-Adaptive Backdoors in Vision Language Modelscited by 0
  11. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  12. 2026ReShift: Aha-Moment-Driven Reasoning-Level Backdoor Attacks on Vision-Language Modelscited by 0
2 more
  1. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  2. 2026FreqDoor: A Hidden Trojan in the Frequency Domain for Backdoor Attacks on Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  2. 2026DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
  3. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
  4. 2025MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models
  5. 2026Test-Time Attention Purification for Backdoored Large Vision Language Models
  6. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models
  7. 2024Invisible Backdoor Attack against Self-supervised Learning
  8. 2026TMT: Runtime Backdoor Detection for Vision-Language-Action Policies on Unseen Tasks