mlsanityData to 7 Oct 2026

VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wild

arXiv 2410.00296 ↗First posted 1 Oct 2024cs.LG cs.CRcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Vision-language Models (VLMs) are essential for contextual understanding of both visual and textual information. However, their vulnerability to adversarially manipulated inputs presents significant risks, leading to compromised outputs and raising concerns about the reliability in VLM-integrated applications. Detecting these malicious prompts is thus crucial for maintaining trust in VLM generations. A major challenge in developing a safeguarding prompt classifier is the lack of a large amount of labeled benign and malicious data. To address the issue, we introduce VLMGuard, a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. These unlabeled prompts, which naturally arise when VLMs are deployed in the open world, consist of both benign and malicious information. To harness the unlabeled data, we present an automated maliciousness estimation score for distinguishing between benign and malicious samples within this unlabeled mixture, thereby enabling the training of a binary prompt classifier on top. Notably, our framework does not require extra human annotations and is robust to realistic prompt variations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that VLMGuard achieves superior detection results, improving AUROC by 5.39% on average over the state-of-the-art method. Disclaimer: This paper may contain offensive examples; reader discretion is advised. Code is available at: https://github.com/radiolab-ntu/vlmguard.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 63 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

40 of 63 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models31 Jul
  2. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Models13 Jun
  3. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models11 Jun
  4. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  5. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  7. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  8. 2024Red Teaming Visual Language Models23 Jan
  9. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  10. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  11. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  11. 2023Detecting Language Model Attacks with Perplexitycited by 411
  12. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
17 more
  1. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  2. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  3. 2023Are aligned neural networks adversarially aligned?cited by 305
  4. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  5. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  7. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  8. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  9. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  10. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  11. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  12. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  13. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  14. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  15. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  16. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63
  17. 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attackscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  2. 2025Safety in Large Reasoning Models: A Surveycited by 58
  3. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  4. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  5. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  6. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  7. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  8. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  9. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 5
  10. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attackscited by 3
  11. 2026WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agentscited by 3
  12. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
5 more
  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  2. 2025SmoothGuard: Defending Multimodal Large Language Models with Noise Perturbation and Clustering Aggregationcited by 2
  3. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 2
  4. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Promptscited by 0
  5. 2026DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  2. 2026Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization
  3. 2025SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
  4. 2024Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images
  5. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models
  6. 2026Detecting Adversarial Images through Response Profiles of Vision-Language Models
  7. 2024Human-Free Automated Prompting for Vision-Language Anomaly Detection: Prompt Optimization with Meta-guiding Prompt Scheme
  8. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap