mlsanityData to 7 Oct 2026

Backdooring Vision-Language Models with Out-Of-Distribution Data

arXiv 2410.01264 ↗First posted 2 Oct 2024cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The emergence of Vision-Language Models (VLMs) represents a significant advancement in integrating computer vision with Large Language Models (LLMs) to generate detailed text descriptions from visual inputs. Despite their growing importance, the security of VLMs, particularly against backdoor attacks, is under explored. Moreover, prior works often assume attackers have access to the original training data, which is often unrealistic. In this paper, we address a more practical and challenging scenario where attackers must rely solely on Out-Of-Distribution (OOD) data. We introduce VLOOD (Backdooring Vision-Language Models with Out-of-Distribution Data), a novel approach with two key contributions: (1) demonstrating backdoor attacks on VLMs in complex image-to-text tasks while minimizing degradation of the original semantics under poisoned inputs, and (2) proposing innovative techniques for backdoor injection without requiring any access to the original training data. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of VLOOD, revealing a critical security vulnerability in VLMs and laying the foundation for future research on securing multimodal models against sophisticated threats.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 44 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

30 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TrojVLM: Backdoor Attack Against Vision Language Models28 Sep
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  3. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  4. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models5 Feb

Other references · 26

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2018Spectral Signatures in Backdoor Attackscited by 457
14 more
  1. 2020Backdoor Learning: A Surveycited by 412
  2. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  3. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  4. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  5. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  6. 2022The "Beatrix'' Resurrections: Robust Backdoor Detection via Gram Matricescited by 62
  7. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  8. 2022Data Poisoning Attacks Against Multimodal Encoderscited by 41
  9. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  10. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 30
  11. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
  12. 2024BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learningcited by 15
  13. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
  14. 2024Object-oriented backdoor attack against image captioningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  3. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  4. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 13
  5. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  6. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  7. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMscited by 9
  8. 2025Credit Risk Identification in Supply Chains Using Generative Adversarial Networkscited by 5
  9. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Modelscited by 5
  10. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5
  11. 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulationcited by 4
  12. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4
19 more
  1. 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
  2. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  3. 2025Concept-Guided Backdoor Attack on Vision Language Modelscited by 2
  4. 2025Backdooring CLIP through Concept Confusioncited by 1
  5. 2025Moir'eXNet: Adaptive Multi-Scale Demoir'eing with Linear Attention Test-Time Training and Truncated Flow Matching Priorcited by 1
  6. 2025VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Modelscited by 1
  7. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  8. 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Modelscited by 0
  9. 2026Phantasia: Context-Adaptive Backdoors in Vision Language Modelscited by 0
  10. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  11. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbationcited by 0
  12. 2026Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Modelscited by 0
  13. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  14. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  15. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  16. 2026Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMscited by 0
  17. 2026Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMscited by 0
  18. 2026FreqDoor: A Hidden Trojan in the Frequency Domain for Backdoor Attacks on Vision-Language Modelscited by 0
  19. 2026Selective Channel Restoration for Backdoored Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
  2. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation
  3. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  4. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  5. 2026Backdoor Learning in Language Models and Vision-Language Models
  6. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  7. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment
  8. 2025MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models