Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
also in 2 more
Abstract (arXiv)
Vision--Language Models (VLMs) are increasingly deployed through a model supply chain in which pretrained checkpoints, architecture definitions, text encoders, and exported computation graphs are distributed by third parties and reused across downstream services. This reuse model creates a security-critical trust boundary: VLM deployments inherit not only learned parameters but also executable behavior encoded in shared model artifacts. In this paper, we show that a malicious provider can exploit this trust boundary by embedding architectural backdoors into VLM supply chains through representation steering. Our attack introduces dormant steering logic into the model architecture through a trigger-gated additive modification of an intermediate representation, without poisoning training data, controlling downstream fine-tuning, or modifying prompts at deployment time. When the trigger is absent, the modification reduces to zero and the model follows its normal computation, preserving clean utility. When the trigger is present, a steering direction shifts the internal representation toward an attacker-defined objective. We evaluate the attack across multiple VLM families and downstream tasks, including visual question answering, text-to-image generation, retrieval, and semantic response biasing. The results show that the proposed architectural steering backdoor compromises integrity, safety enforcement, and ranking fairness while preserving normal behavior on clean inputs. We further show that shared VLM artifacts can carry dormant steering logic against downstream services, and we propose an auditing defense that inspects the executable logic distributed with model artifacts rather than only their learned weights.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 69 references that are in the map. First posted on arXiv 28 Jul 2026.
Builds on
48 of 69 references.
Closest ancestors · 4
Same direction, newest first
- 2026Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization7 Apr
- 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift27 Jun
- 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
- 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP26 Nov
Other references · 44
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023Visual Instruction Tuningcited by 7,869
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2025Qwen3-VL Technical Reportcited by 2,793
32 more
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2019Towards VQA Models That Can Readcited by 1,718
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
- 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
- 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
- 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
- 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 242
- 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
- 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 194
- 2021Poisoning and Backdooring Contrastive Learningcited by 151
- 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
- 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
- 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoningcited by 92
- 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68
- 2025FineVision: Open Data Is All You Needcited by 53
- 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 37
- 2024Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Modelscited by 12
- 2024Architectural Neural Backdoors from First Principlescited by 11
- 2024T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognitioncited by 11
- 2023HOD: A Benchmark Dataset for Harmful Object Detectioncited by 10
- 2024DiffGuard: Text-Based Safety Checker for Diffusion Modelscited by 10
- 2024UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Modelscited by 9
- 2023Universal Backdoor Attackscited by 7
- 2025The Ripple Effect: On Unforeseen Complications of Backdoor Attackscited by 1
- 2025Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defensecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
- 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuning
- 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
- 2026TrustVLA: Mechanism-Guided Inference-Time Defense Against Vision-Language-Action Backdoors
- 2024Backdooring Vision-Language Models with Out-Of-Distribution Data
- 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment
- 2026Directional Embedding Smoothing for Robust Vision Language Models
- 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models