mlsanityData to 7 Oct 2026

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs

arXiv 2402.03757 ↗First posted 6 Feb 2024cs.CV cs.CL cs.LGcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large language models (LLMs) have recently experienced remarkable progress, where the advent of multi-modal large language models (MLLMs) has endowed LLMs with visual capabilities, leading to impressive performances in various multi-modal tasks. However, those powerful MLLMs such as GPT-4V still fail spectacularly when presented with certain image and text inputs. In this paper, we identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers, causing MLLMs to suffer from visual illusion. To quantify the effect, we propose CorrelationQA, the first benchmark that assesses the visual illusion level given spurious images. This benchmark contains 7,308 text-image pairs across 13 categories. Based on the proposed CorrelationQA, we conduct a thorough analysis on 9 mainstream MLLMs, illustrating that they universally suffer from this instinctive bias to varying degrees. We hope that our curated benchmark and evaluation results aid in better assessments of the MLLMs' robustness in the presence of misleading images. The code and datasets are available at https://github.com/MasaiahHan/CorrelationQA.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 30 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

23 of 30 references.

References · 23

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
11 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  4. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  7. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  8. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  9. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  10. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  11. 2020Thinking Fast and Slow in AIcited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  2. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 30
  3. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  4. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  5. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
  6. 2025Beyond Words and Pixels: A Benchmark for Implicit World Knowledge Reasoning in Generative Modelscited by 4
  7. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMscited by 3
  8. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  9. 2026Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewardscited by 1
  10. 2026MotionHalluc: Diagnosing Kinematic Hallucinations in Fine-Grained Motion Reasoningcited by 0
  11. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs
  2. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
  3. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  4. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  5. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation
  6. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
  7. 2026SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions
  8. 2026V-FAT: Benchmarking Visual Fidelity Against Text-bias