mlsanityData to 14 Oct 2026

Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?

arXiv 2506.18322 ↗First posted 23 Jun 2025cs.CV cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Finetuning can cause spurious correlations to arise between non-essential features and the target labels, but benchmarks to study these effects involve contrived settings and narrow tasks. In contrast, we consider spurious correlations in multi-modal Large Vision Language Models (LVLMs) pretrained on extensive and diverse datasets without explicit task supervision. We develop a benchmark by sourcing GPT-4o errors on real-world visual-question-answering (VQA) benchmarks, then curating a subset through LVLM-human annotation and synthetic counterfactual evaluation to identify errors caused by spurious correlations. This process yields SpuriVerse, a novel benchmark comprised of 124 distinct types of spurious correlations extracted from real-world datasets, each containing 1 realistic and 10 synthetic VQA samples for a total of 1364 multiple choice questions. We evaluate 15 open and closed-source LVLMs on SpuriVerse, finding that even state-of-the-art closed-source models struggle significantly, achieving at best only 37.1% accuracy. Fine-tuning on synthetic examples that emphasize the spurious correlation improves performance to 78.40%, suggesting that training on diverse spurious patterns generalizes to unseen situations: models appear to learn to avoid "shortcuts" and attend to the overall image context.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

31 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  11. 2019Invariant Risk Minimizationcited by 1,854
  12. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
18 more
  1. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  6. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  7. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  8. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  9. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 209
  10. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  11. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  12. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  13. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  14. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  15. 2018Feature-Wise Bias Amplificationcited by 31
  16. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 16
  17. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 6
  18. 2023InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 3
  2. 2026Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specificationcited by 3
  3. 2025ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Modelscited by 2
  4. 2025Unbiased Visual Reasoning with Controlled Visual Inputscited by 2
  5. 2026MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtrainingcited by 2
  6. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  7. 2026Can AI Agents Synthesize Scientific Conclusions?cited by 1
  8. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalitiescited by 1
  9. 2026Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoningcited by 1
  10. 2026InjectFlow: Weak Guides Strong via Orthogonal Injection for Flow Matchingcited by 0
  11. 2026Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updatescited by 0
  12. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0
2 more
  1. 2026When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMscited by 0
  2. 2026Spurious Tool Use: When RL Agents Learn the Wrong Reason to Actcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs
  2. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  3. 2024RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
  4. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  5. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  6. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  7. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
  8. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models