ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
also in 1 more
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities. While prior works have demonstrated the feasibility of backdoors in MLLMs via fine-tuning data poisoning to manipulate inference, the underlying mechanisms of backdoor attacks remain opaque, complicating the understanding and mitigation. To bridge this gap, we propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors. We first establish that normal downstream task alignment--even when restricted to projector fine--tuning--introduces vulnerability to backdoor injection, whose activation mechanism is different from that observed in text-only LLMs. Through extensive experiments across four backdoor variants, we uncover:(1) Low-Rank Structure: Backdoor injection updates appear overall full-rank and lack dedicated ``trigger neurons'', but the backdoor-critical parameters are encoded within a low-rank subspace of the projector;(2) Activation Mechanism: Both clean and poisoned embedding undergoes a semantic shift toward a shared direction aligned with the backdoor target, but the shifting magnitude scales linearly with the input norm, resulting in the distinct backdoor activation on poisoned samples. Our code is available at: https://anonymous.4open.science/r/ProjLens-8FD7
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 72 references that are in the map. First posted on arXiv 21 Apr 2026.
Builds on
58 of 72 references.
Closest ancestors · 10
Same direction, newest first
- 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models24 Nov
- 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation8 Jun
- 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models26 May
- 2025Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving9 May
- 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves29 Oct
- 2024Backdooring Vision-Language Models with Out-Of-Distribution Data2 Oct
- 2024TrojVLM: Backdoor Attack Against Vision Language Models28 Sep
- 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
- 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
- 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models5 Feb
Other references · 48
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023Visual Instruction Tuningcited by 7,792
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2025Qwen2.5-Omni Technical Reportcited by 704
36 more
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
- 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
- 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
- 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
- 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
- 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
- 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
- 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
- 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
- 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 80
- 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
- 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
- 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 63
- 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
- 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
- 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
- 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
- 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
- 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
- 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 30
- 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
- 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMscited by 22
- 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
- 2024Q-Probe: A Lightweight Approach to Reward Maximization for Language Modelscited by 15
- 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
- 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
- 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 8
- 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Modelscited by 7
- 2025FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challengingcited by 6
- 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 5
- 2025Visually Interpretable Subtask Reasoning for Visual Question Answeringcited by 4
- 2025Probing Neural Combinatorial Optimization Modelscited by 3
- 2025Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models
- 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
- 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
- 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuning
- 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
- 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models
- 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
- 2025Robust Anti-Backdoor Instruction Tuning in LVLMs