mlsanityData to 7 Oct 2026

How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

arXiv 2402.13220 ↗First posted 20 Feb 2024cs.CV cs.CLcited by 47

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The remarkable advancements in Multimodal Large Language Models (MLLMs) have not rendered them immune to challenges, particularly in the context of handling deceptive information in prompts, thus producing hallucinated responses under such conditions. To quantitatively assess this vulnerability, we present MAD-Bench, a carefully curated benchmark that contains 1000 test samples divided into 5 categories, such as non-existent objects, count of objects, and spatial relationship. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4v, Reka, Gemini-Pro, to open-sourced models, such as LLaVA-NeXT and MiniCPM-Llama3. Empirically, we observe significant performance gaps between GPT-4o and other models; and previous robust instruction-tuned models are not effective on this new benchmark. While GPT-4o achieves 82.82% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 9% to 50%. We further propose a remedy that adds an additional paragraph to the deceptive prompts to encourage models to think twice before answering the question. Surprisingly, this simple method can even double the accuracy; however, the absolute numbers are still too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark to stimulate further research to enhance model resilience against deceptive prompts.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 82 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

65 of 82 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023VIGC: Visual Instruction Generation and Correction24 Aug
  9. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
4 more
  1. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  2. 2023Generating Images with Multimodal Language Models26 May
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
37 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  13. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  14. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  15. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  16. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  17. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  18. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  19. 2022Prompting GPT-3 To Be Reliablecited by 258
  20. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  21. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  22. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  23. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  24. 2023Fine-tuning Language Models for Factualitycited by 219
  25. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  26. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  27. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  28. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  29. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  30. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  31. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  32. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  33. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  34. 2023Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Modelscited by 36
  35. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  36. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  37. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 16; 2026: 6.

47 papers.

Sort:

By area:

  1. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  2. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  3. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  4. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  5. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 41
  6. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  7. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 30
  8. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
  9. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  10. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  11. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  12. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?cited by 24
35 more
  1. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 22
  2. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  3. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Frameworkcited by 17
  4. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  5. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14
  6. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  7. 2024PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysiscited by 13
  8. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  9. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)cited by 10
  10. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  11. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Modelscited by 8
  12. 2024Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuningcited by 7
  13. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  14. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  15. 2025A Taxonomy of Prompt Defects in LLM Systemscited by 7
  16. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7
  17. 2024Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructionscited by 6
  18. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 6
  19. 2025EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glassescited by 6
  20. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 5
  21. 2024CableInspect-AD: An Expert-Annotated Anomaly Detection Datasetcited by 5
  22. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  23. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Modelscited by 4
  24. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approachcited by 3
  25. 2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modelingcited by 3
  26. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  27. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  28. 2025Judge Before Answer: Can MLLM Discern the False Premise in Question?cited by 2
  29. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 2
  30. 2025Exploring and Mitigating Fawning Hallucinations in Large Language Modelscited by 1
  31. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 1
  32. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  33. 2025MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAGcited by 0
  34. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  35. 2026Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?
  2. 2025Hidden in Plain Sight: Evaluation of the Deception Detection Capabilities of LLMs in Multimodal Settings
  3. 2025Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs
  4. 2025Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Models
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  6. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
  7. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  8. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions