mlsanityData to 7 Oct 2026

Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift

arXiv 2406.18844 ↗First posted 27 Jun 2024cs.CVcited by 45

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Instruction tuning enhances large vision-language models (LVLMs) but increases their vulnerability to backdoor attacks due to their open design. Unlike prior studies in static settings, this paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains. We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness under visual and text domain shifts. Our findings reveal two insights: (1) backdoor generalizability improves when distinctive trigger patterns are independent of specific data domains or model architectures, and (2) the competitive interaction between trigger patterns and clean semantic regions, where guiding the model to predict triggers enhances attack generalizability. Based on these insights, we propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas using attributional interpretation. Experiments with OpenFlamingo, Blip-2, and Otter show that MABA significantly boosts the attack success rate of generalization by 36.4%, achieving a 97% success rate at a 0.2% poisoning rate. This study reveals limitations in current evaluations and highlights how enhanced backdoor generalizability poses a security threat to LVLMs, even without test data access.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 68 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

49 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  3. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
34 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  9. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  12. 2020Input-Aware Dynamic Backdoor Attackcited by 304
  13. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  14. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  15. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  16. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspectivecited by 155
  17. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  18. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 112
  19. 2023On the Exploitability of Instruction Tuningcited by 110
  20. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  21. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  22. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  23. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 35
  24. 2024Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detectioncited by 34
  25. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  26. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 30
  27. 2024Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attackcited by 26
  28. 2024Towards Robust Physical-world Backdoor Attacks on Lane Detectioncited by 25
  29. 2024Can We Talk Models Into Seeing the World Differently?cited by 23
  30. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21
  31. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 19
  32. 2022A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation Is the Fixed Point of Adversarial Gamecited by 16
  33. 2021Poisoning Attack against Estimating from Pairwise Comparisonscited by 14
  34. 2024Environmental Matching Attack Against Unmanned Aerial Vehicles Object Detectioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 21; 2026: 20.

45 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  5. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  6. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  7. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  8. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  9. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  10. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  11. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  12. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuningcited by 7
33 more
  1. 2024CopyrightShield: Enhancing Diffusion Model Security against Copyright Infringement Attackscited by 5
  2. 2024TrapFlow: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learningcited by 5
  3. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  4. 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Modelscited by 5
  5. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Modelscited by 5
  6. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehiclescited by 4
  7. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  8. 2025Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignmentcited by 3
  9. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  10. 2026CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Modelscited by 3
  11. 2026Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothingcited by 3
  12. 2025PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generationcited by 2
  13. 2025Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Modelscited by 2
  14. 2025RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulationcited by 2
  15. 2025Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivativescited by 2
  16. 2026BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learningcited by 2
  17. 2026BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviationcited by 1
  18. 2026Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggerscited by 1
  19. 2026Distillation Traps and Guards: A Calibration Knob for LLM Distillabilitycited by 1
  20. 2025Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Modelscited by 0
  21. 2025When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTIcited by 0
  22. 2026TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuningcited by 0
  23. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  24. 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Modelscited by 0
  25. 2026Phantasia: Context-Adaptive Backdoors in Vision Language Modelscited by 0
  26. 2026Beyond Attack Success Rate: Examining Trigger Leakage in Vision-Language Agentic Systemscited by 0
  27. 2026Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steeringcited by 0
  28. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  29. 2026SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defensecited by 0
  30. 2026Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMscited by 0
  31. 2026Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMscited by 0
  32. 2026TrojanWorld: Backdooring World-Model Agents via Imagination Steeringcited by 0
  33. 2026Selective Channel Restoration for Backdoored Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models
  2. 2025Robust Anti-Backdoor Instruction Tuning in LVLMs
  3. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
  4. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation
  5. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks
  6. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment
  7. 2025Concept-Guided Backdoor Attack on Vision Language Models
  8. 2026EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy