mlsanityData to 7 Oct 2026

MirrorCheck: Efficient Adversarial Defense for Vision-Language Models

arXiv 2406.09250 ↗First posted 13 Jun 2024cs.CV cs.AI cs.LGcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly susceptible to sophisticated adversarial attacks, including adaptive strategies specifically designed to bypass existing defenses. To address this vulnerability, we propose MirrorCheck, a robust and model-agnostic detection framework that operates effectively in both unimodal and multimodal settings. MirrorCheck leverages Text-to-Image (T2I) models to regenerate visual content from captions produced by the target model and assesses semantic consistency by comparing feature-space embeddings between the original and synthesized images. To enhance robustness against adaptive attacks, MirrorCheck introduces a stochastic defense strategy that randomly selects T2I generators and image encoders from a diverse model zoo. Additionally, we incorporate a novel One-Time-Use (OTU) perturbation applied to the selected encoder embeddings, regulated by a scaling factor, which decreases the effectiveness of adaptive attacks. Extensive experiments across multiple threat scenarios demonstrate that MirrorCheck consistently outperforms baseline methods, and maintains its utility even under strong adaptive adversarial conditions.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 99 of its 122 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

99 of 122 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors17 May
  2. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2016Adversarial examples in the physical worldcited by 1,762
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  16. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  17. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  18. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  19. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  20. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  21. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2016Adversarial Machine Learning at Scalecited by 934
  24. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  25. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  26. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  27. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  28. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  29. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  30. 2019Adversarial Training for Free!cited by 638
  31. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  32. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  33. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  34. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  35. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  36. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
  37. 2022Diffusion Models for Adversarial Purificationcited by 469
  38. 2015Importance Weighted Autoencoderscited by 454

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  4. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  5. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  6. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  7. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  8. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  9. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  10. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Modelscited by 3
  11. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  12. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMscited by 2
7 more
  1. 2025Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Modelscited by 1
  2. 2026BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviationcited by 1
  3. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  4. 2026NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Drivingcited by 0
  5. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Promptscited by 0
  6. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correctioncited by 0
  7. 2026Securing Multimodal AI through Internal Information Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  3. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
  4. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
  5. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  6. 2026Adversarial Defense in Vision-Language Models: An Overview
  7. 2026VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defense
  8. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships