mlsanityData to 7 Oct 2026

LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models

arXiv 2502.12359 ↗First posted 17 Feb 2025cs.CVcited by 14

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have shown impressive performance in various tasks. However, LVLMs suffer from hallucination, which hinders their adoption in the real world. Existing studies emphasized that the strong language priors of LVLMs can overpower visual information, causing hallucinations. However, the positive role of language priors is the key to a powerful LVLM. If the language priors are too weak, LVLMs will struggle to leverage rich parameter knowledge and instruction understanding abilities to complete tasks in challenging visual scenarios where visual information alone is insufficient. Therefore, we propose a benchmark called LanP to rethink the impact of Language Priors in LVLMs. It is designed to investigate how strong language priors are in current LVLMs. LanP consists of 170 images and 340 corresponding well-designed questions. Extensive experiments on 25 popular LVLMs reveal that many LVLMs' language priors are not strong enough to effectively aid question answering when objects are partially hidden. Many models, including GPT-4 Turbo, exhibit an accuracy below 0.5 in such a scenario.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 54 of its 64 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

54 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning30 Aug

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
38 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  13. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  14. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  15. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  16. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  17. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  18. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  19. 2018Getting to Know Low-light Images with The Exclusively Dark Datasetcited by 157
  20. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  21. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  22. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  23. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  24. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 82
  25. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  26. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  27. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  28. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  29. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  30. 2019DAVANet: Stereo Deblurring with View Aggregationcited by 37
  31. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36
  32. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  33. 2023Leveraging Vision-Language Models for Granular Market Change Predictioncited by 22
  34. 2024Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotationcited by 21
  35. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 19
  36. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  37. 2024LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learningcited by 17
  38. 2024Law of Vision Representation in MLLMscited by 16

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  2. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  3. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximizationcited by 12
  4. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 9
  5. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  6. 2025How Far are LLMs from Real Search? A Comprehensive Study on Efficiency, Completeness, and Inherent Capabilitiescited by 4
  7. 2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
  8. 2025Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Modelscited by 3
  9. 2026CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoningcited by 3
  10. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  11. 2025Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Modelscited by 0
  12. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMscited by 0
2 more
  1. 2026Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactionscited by 0
  2. 2026Does Local Video Understanding Transfer Across Encounters? The EgoGears Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
  2. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models
  3. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  4. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  5. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  6. 2026Diagnosing Visual Ignorance in Vision-Language Models
  7. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models