mlsanityData to 14 Oct 2026

How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

arXiv 2311.16101 ↗First posted 27 Nov 2023cs.CV cs.CL cs.LGcited by 108

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness. For the OOD evaluation, we present two novel VQA datasets, each with one variant, designed to test model performance under challenging conditions. In exploring adversarial robustness, we propose a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses. Moreover, we assess the efficacy of two jailbreaking strategies, targeting either the vision or language component of VLLMs. Our evaluation of 21 diverse models, ranging from open-source VLLMs to GPT-4V, yields interesting observations: 1) Current VLLMs struggle with OOD texts but not images, unless the visual information is limited; and 2) These VLLMs can be easily misled by deceiving vision encoders only, and their vision-language training often compromise safety protocols. We release this safety evaluation suite at https://github.com/UCSC-VLAA/vllm-safety-benchmark.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 60 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

49 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
35 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  9. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  10. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  12. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  13. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  14. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  15. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 183
  16. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  17. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  18. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  19. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  20. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  21. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  22. 2018SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesiscited by 81
  23. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
  24. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  25. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  26. 2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
  27. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  28. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  29. 2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
  30. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  31. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  32. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14
  33. 2023ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oraclescited by 5
  34. 2023ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialoguecited by 3
  35. 2023AGIBench: A Multi-granularity, Multimodal, Human-referenced, Auto-scoring Benchmark for Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 55; 2025: 42; 2026: 11.

108 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  2. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  3. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  5. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  6. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  7. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  8. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  9. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  10. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  11. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  12. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
38 more
  1. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  2. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  3. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  6. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  7. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  8. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 68
  9. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  10. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  11. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 60
  12. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 57
  13. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  14. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  15. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  16. 2025FineVision: Open Data Is All You Needcited by 53
  17. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  18. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 40
  19. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 39
  20. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  21. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 36
  22. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  23. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 34
  24. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  25. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 33
  26. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  27. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  28. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 33
  29. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 32
  30. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 32
  31. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  32. 2024MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?cited by 30
  33. 2024Safety Alignment for Vision Language Modelscited by 28
  34. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  35. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  36. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  37. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  38. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Modelcited by 26

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models
  3. 2026A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
  4. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  5. 2025A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applications
  6. 2025VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
  7. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  8. 2024Retention Score: Quantifying Jailbreak Risks for Vision Language Models