mlsanityData to 14 Oct 2026

SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs

arXiv 2503.08884 ↗First posted 11 Mar 2025cs.CV cs.CL cs.LGcited by 7

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Unimodal vision models are known to rely on spurious correlations, but it remains unclear to what extent Multimodal Large Language Models (MLLMs) exhibit similar biases despite language supervision. In this paper, we investigate spurious bias in MLLMs and introduce SpurLens, a pipeline that leverages GPT-4 and open-set object detectors to automatically identify spurious visual cues without human supervision. Our findings reveal that spurious correlations cause two major failure modes in MLLMs: (1) over-reliance on spurious cues for object recognition, where removing these cues reduces accuracy, and (2) object hallucination, where spurious cues amplify the hallucination by over 10x. We validate our findings in various MLLMs and datasets. Beyond diagnosing these failures, we explore potential mitigation strategies, such as prompt ensembling and reasoning-based prompting, and conduct ablation studies to examine the root causes of spurious bias in MLLMs. By exposing the persistence of spurious correlations, our study calls for more rigorous evaluation methods and mitigation strategies to enhance the reliability of MLLMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 45 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

25 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio16 Oct
  2. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 20

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  10. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  11. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 343
8 more
  1. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  2. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  3. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  4. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  5. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  6. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  7. 2024RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Modelscited by 20
  8. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Drivingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  2. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  3. 2025Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routingcited by 5
  4. 2026GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodescited by 1
  5. 2026UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifierscited by 0
  6. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0
  7. 2026SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Cameracited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
  2. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
  3. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  4. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  5. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models
  6. 2025Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention
  7. 2024Hallucination of Multimodal Large Language Models: A Survey
  8. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models