mlsanityData to 7 Oct 2026

Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability

arXiv 2501.01346 ↗First posted 2 Jan 2025cs.CV cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and textual representations is not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challenges at multiple levels: the data level, the model level, and the inference level. We provide a comprehensive review of existing mitigation strategies, categorizing them into parameter-frozen and parameter-tuning approaches. Finally, we outline promising future research directions, emphasizing the need for standardized evaluation protocols and in-depth explainability studies.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 90 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

67 of 90 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective1 Oct
  2. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuning19 Apr
  3. 2023MAFA: Managing False Negatives for Vision-Language Pre-training11 Dec
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  6. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
38 more
  1. 2018Object Hallucination in Image Captioningcited by 661
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2019On Variational Bounds of Mutual Informationcited by 550
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  9. 2023Multimodal Deep Learningcited by 257
  10. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  11. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  12. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  13. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  14. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  15. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  16. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  17. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  18. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  19. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  20. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  21. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  22. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  23. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  24. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  25. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  26. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  27. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  28. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  29. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  30. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  31. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  32. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  33. 2024A Vision Check-up for Language Modelscited by 36
  34. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  35. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  36. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  37. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  38. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentationcited by 27

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 12.

28 papers.

Sort:

By area:

  1. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  2. 2025MatterChat: A Multi-Modal LLM for Material Sciencecited by 18
  3. 2025Robust Multimodal Large Language Models Against Modality Conflictcited by 17
  4. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  5. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Setcited by 12
  6. 2025MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMscited by 9
  7. 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
  8. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  9. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMscited by 7
  10. 2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimizationcited by 6
  11. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  12. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
16 more
  1. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  2. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2
  3. 2026VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Modelscited by 2
  4. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Modelscited by 1
  5. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  6. 2025CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updatescited by 1
  7. 2026Finding Distributed Object-Centric Properties in Self-Supervised Transformerscited by 1
  8. 2026Breaking the Illusion: When Positive Meets Negative in Multimodal Decodingcited by 1
  9. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  10. 2026One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Modelscited by 1
  11. 2025PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifierscited by 0
  12. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  13. 2026Can Large Language Models Make Everyone Happy?cited by 0
  14. 2026MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Modelscited by 0
  15. 2026ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?cited by 0
  16. 2026Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge
  2. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment
  3. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  4. 2025Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models
  5. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  6. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
  7. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  8. 2025LVLM-Aided Alignment of Task-Specific Vision Models