mlsanityData to 14 Oct 2026

Robust image classification with multi-modal large language models

arXiv 2412.10353 ↗First posted 13 Dec 2024cs.CV cs.CR cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully crafted input samples that can cause models to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. However, most of these approaches focus on a single data modality, overlooking the relationships between visual patterns and textual descriptions of the input. In this paper, we propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information to further enhance their robustness. MultiShield leverages multi-modal large language models to detect adversarial examples and abstain from uncertain classifications when there is no alignment between textual and visual representations of the input. Extensive evaluations on CIFAR-10 and ImageNet datasets, using robust and non-robust image classification models, demonstrate that MultiShield can be easily integrated to detect and reject adversarial examples, outperforming the original defenses.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 50 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

37 of 50 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  11. 2023A Survey on Multimodal Large Language Modelscited by 909
  12. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
25 more
  1. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  2. 2019Adversarial Training for Free!cited by 643
  3. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  4. 2015Towards Open Set Deep Networkscited by 538
  5. 2019On Evaluating Adversarial Robustnesscited by 447
  6. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  7. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 414
  8. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  9. 2021Improving Robustness using Generated Datacited by 234
  10. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  11. 2018Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learningcited by 207
  12. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  13. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  14. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  15. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  16. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  17. 2017SafetyNet: Detecting and Rejecting Adversarial Examples Robustlycited by 76
  18. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  19. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  20. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  21. 2023Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustnesscited by 36
  22. 2019Deep Neural Rejection against Adversarial Examplescited by 32
  23. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 24
  24. 2017Is Deep Learning Safe for Robot Vision? Adversarial Examples against the iCub Humanoidcited by 19
  25. 2024AttackBench: Evaluating Gradient-based Attacks for Adversarial Examplescited by 14

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  2. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  2. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  3. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
  4. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
  5. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks
  6. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
  7. 2025Coordinated Robustness Evaluation Framework for Vision-Language Models
  8. 2025Robust Defense Strategies for Multimodal Contrastive Learning: Efficient Fine-tuning Against Backdoor Attacks