mlsanityData to 7 Oct 2026

Probing Visual Language Priors in VLMs

arXiv 2501.00569 ↗First posted 31 Dec 2024cs.CV cs.LGcited by 44

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Despite recent advances in Vision-Language Models (VLMs), they may over-rely on visual language priors existing in their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although, humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4 achieves only 66.17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training. Our training objectives compel VLMs to focus more on the actual visual inputs, and we demonstrate their effectiveness in boosting the performance of open-source VLMs, including LLaVA-v1.5 and Cambrian.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 76 of its 97 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

76 of 97 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement24 May
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  14. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  19. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  20. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  21. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  22. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  24. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  26. 2018Object Hallucination in Image Captioningcited by 661
  27. 2024Self-Rewarding Language Modelscited by 654
  28. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  29. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  30. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  31. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  32. 2021InfographicVQAcited by 430
  33. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  34. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  35. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  36. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  37. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  38. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 35.

44 papers.

Sort:

By area:

  1. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  2. 2025Vision Language Models are Biasedcited by 63
  3. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  4. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  5. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  6. 2026Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflictscited by 7
  7. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmarkcited by 7
  8. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  9. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 6
  10. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  11. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  12. 2026Visual Preference Optimization with Rubric Rewardscited by 5
32 more
  1. 2025VISaGE: Understanding Visual Generics and Exceptionscited by 4
  2. 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillationcited by 4
  3. 2026Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluationcited by 3
  4. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMscited by 2
  5. 2026Diagnosing Visual Ignorance in Vision-Language Modelscited by 2
  6. 2025Language-Guided Invariance Probing of Vision-Language Modelscited by 1
  7. 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Frameworkcited by 1
  8. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  9. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  10. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  11. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  12. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  13. 2026Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Contextually Important Momentscited by 0
  14. 2026Auditing Disability Representation in Vision-Language Modelscited by 0
  15. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 0
  16. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  17. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  18. 2026Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimizationcited by 0
  19. 2026See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RLcited by 0
  20. 2026Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?cited by 0
  21. 2026Prior Bias in Vision Language Models on UML Diagram Interpretationcited by 0
  22. 2026How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQAcited by 0
  23. 2026Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMscited by 0
  24. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  25. 2026Prior Directions: Why GUI Grounding Gets Locked in the Pastcited by 0
  26. 2026It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scalingcited by 0
  27. 2026SABRE: Scalable and Automated Benchmarking of VLMs under Stresscited by 0
  28. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Actcited by 0
  29. 2026CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Modelscited by 0
  30. 2026MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Modelscited by 0
  31. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0
  32. 2026When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  2. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  3. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  4. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  6. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models