mlsanityData to 7 Oct 2026

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

arXiv 2403.18346 ↗First posted 27 Mar 2024cs.CL cs.CVcited by 39

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have facilitated the development of Multimodal LLMs (MLLMs). Despite their impressive capabilities, MLLMs often suffer from over-reliance on unimodal biases (e.g., language bias and vision bias), leading to incorrect answers or hallucinations in complex multimodal tasks. To investigate this issue, we propose a causal framework to interpret the biases in Visual Question Answering (VQA) problems. Within this framework, we conduct an in-depth causal analysis to assess the causal effect of these biases on MLLM predictions. Based on the analysis, we introduce 1) a novel MORE dataset with 12,000 challenging VQA instances requiring multi-hop reasoning and overcoming unimodal biases. 2) a causality-enhanced agent framework CAVE that guides models to comprehensively integrate information from different modalities and mitigate biases. Our experiments show that MLLMs perform poorly on MORE, indicating strong unimodal biases and limited semantic understanding. However, when integrated with our CAVE, promising improvements in reasoning and bias mitigation can be seen. These findings provide important insights for the development of more robust MLLMs and contribute to the broader goal of advancing multimodal AI systems capable of deeper understanding and reasoning. Our project page is at https://github.com/OpenCausaLab/MORE.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 57 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

35 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities26 Jan
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  3. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
18 more
  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  2. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  3. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  4. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  5. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  7. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  8. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  9. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  10. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  11. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  12. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  13. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  14. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  15. 2019PaperRobot: Incremental Draft Generation of Scientific Ideascited by 37
  16. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  17. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  18. 2023Improving Large Language Models in Event Relation Logical Predictioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 12.

39 papers.

Sort:

By area:

  1. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  2. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  3. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  4. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  5. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2025Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcutscited by 26
  8. 2024Causal Evaluation of Language Modelscited by 18
  9. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  10. 2024CLEAR: Can Language Models Really Understand Causal Graphs?cited by 14
  11. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  12. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
27 more
  1. 2025Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Imagescited by 12
  2. 2024CELLO: Causal Evaluation of Large Vision-Language Modelscited by 10
  3. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  4. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
  5. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  6. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  7. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstentioncited by 6
  8. 2024Measuring Agreeableness Bias in Multimodal Modelscited by 4
  9. 2025Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generationcited by 4
  10. 2024Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Abilitycited by 3
  11. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  12. 2026Moral Sycophancy in Vision Language Modelscited by 3
  13. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3
  14. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  15. 2026Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMscited by 2
  16. 2024More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Modelscited by 1
  17. 2025Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarkscited by 1
  18. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainabilitycited by 1
  19. 2025Mitigating Hidden Confounding by Progressive Confounder Imputation via Large Language Modelscited by 0
  20. 2025A Video Is Not Worth a Thousand Wordscited by 0
  21. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approachcited by 0
  22. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  23. 2026Demographic and Linguistic Bias Evaluation in Omnimodal Language Modelscited by 0
  24. 2026Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Modelscited by 0
  25. 2026Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosiscited by 0
  26. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  27. 2026From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Makingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA
  2. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
  3. 2024Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities
  4. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  5. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  6. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  7. 2025Evaluating Variance in Visual Question Answering Benchmarks
  8. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals