mlsanityData to 7 Oct 2026

VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts

arXiv 2604.06502 ↗First posted 7 Apr 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integration. Existing defenses suffer from efficiency and robustness. To address these challenges, we first propose the Multimodal Aggregated Feature Extraction (MAFE) framework that enables CLIP to handle long text and fuse multimodal information into unified representations. Through empirical analysis of MAFE-extracted features, we discover distinct distributional patterns between benign and malicious prompts. Building upon this finding, we develop VLMShield, a lightweight safety detector that efficiently identifies multimodal malicious attacks as a plug-and-play solution. Extensive experiments demonstrate superior performance across multiple dimensions, including robustness, efficiency, and utility. Through our work, we hope to pave the way for more secure multimodal AI deployment. Code is available at [this https URL](https://github.com/pgqihere/VLMShield).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

22 of 28 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  2. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wild1 Oct
  3. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Models13 Jun
  4. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  5. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  7. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  8. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  9. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  10. 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attackscited by 46
  11. 2024What to align in multimodal contrastive learning?cited by 41
  12. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
3 more
  1. 2023VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Modelscited by 9
  2. 2024GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Modelscited by 9
  3. 2025SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
  2. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  3. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
  4. 2026Breaking the weakest link to evade vision language models
  5. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  6. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization
  7. 2024Backdooring Vision-Language Models with Out-Of-Distribution Data
  8. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks