mlsanityData to 7 Oct 2026

Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance

arXiv 2411.14279 ↗First posted 21 Nov 2024cs.CV cs.CLcited by 32

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. However, despite showing promising performance, LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We identify two primary reasons for this bias: 1. Different scales of training data between the pretraining stage of LLM and multimodal alignment stage. 2. The learned inference bias due to short-term dependency of text data. Therefore, we propose LACING, a systemic framework designed to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG). Specifically, MDA introduces a parallel dual-attention mechanism that enhances the integration of visual inputs across the model. IFG introduces a learnable soft visual prompt during training and inference to replace visual inputs, designed to compel LVLMs to prioritize text inputs. Then, IFG further proposes a novel decoding strategy using the soft visual prompt to mitigate the model's over-reliance on adjacent text inputs. Comprehensive experiments demonstrate that our method effectively debiases LVLMs from their language bias, enhancing visual comprehension and reducing hallucinations without requiring additional training resources or data. The code and model are available at [lacing-lvlm.github.io](https://lacing-lvlm.github.io).

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 73 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

42 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  4. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2019Towards VQA Models That Can Readcited by 1,696
20 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  12. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  13. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  14. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  15. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  16. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  17. 2017Frustratingly Short Attention Spans in Neural Language Modelingcited by 18
  18. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 18
  19. 2024A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generationcited by 9
  20. 2024Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 17.

32 papers.

Sort:

By area:

  1. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  2. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  3. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  4. 2025A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Taskscited by 17
  5. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lenscited by 8
  6. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  7. 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
  8. 2025Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routingcited by 5
  9. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
  10. 2026Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillationcited by 4
  11. 2025SafeCoT: Improving VLM Safety with Minimal Reasoningcited by 3
  12. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 3
20 more
  1. 2025From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decompositioncited by 3
  2. 2026PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMscited by 3
  3. 2026Moral Sycophancy in Vision Language Modelscited by 3
  4. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gaincited by 3
  5. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3
  6. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  7. 2025MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Modelscited by 1
  8. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomescited by 1
  9. 2025Suppressing VLM Hallucinations with Spectral Representation Filteringcited by 1
  10. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  11. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editingcited by 1
  12. 2026ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMscited by 1
  13. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  14. 2025Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Modelscited by 0
  15. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 0
  16. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  17. 2026Wiener Representation Filtering for VLM Hallucination Suppressioncited by 0
  18. 2026Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agentscited by 0
  19. 2026How do LLMs Evaluate Perceived Moral Agency? Investigating Moral Decision-Making in Human-Artificial Agents Interactionscited by 0
  20. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training
  2. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  3. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  4. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  5. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  6. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  7. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models
  8. 2024Optimizing Vision-Language Interactions Through Decoder-Only Models