mlsanityData to 14 Oct 2026

Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization

arXiv 2503.17928 ↗First posted 23 Mar 2025cs.CV cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models excel in various tasks, yet often struggle with modality bias, where the model tends to rely heavily on a single modality and overlook critical information in other modalities, which leads to incorrect focus and generating irrelevant responses. In this paper, we propose using the paradigm of preference optimization to solve the modality bias problem, including RLAIFVBias, a debiased preference optimization dataset, and a Noise Aware Preference Optimization algorithm. Specifically, we first construct the dataset by introducing perturbations to reduce the informational content of certain modalities, compelling the model to rely on a specific modality when generating negative responses. To address the inevitable noise in automatically constructed data, we combine the noise robust Mean Absolute Error with the Binary Cross Entropy in Direct Preference Optimization by a negative Box Cox transformation, and dynamically adjust the algorithm noise robustness based on the evaluated noise levels in the data. Extensive experiments validate our approach, demonstrating not only its effectiveness in mitigating modality bias but also its significant role in minimizing hallucinations.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 66 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

56 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar
  2. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 52

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
38 more
  1. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  4. 2018Object Hallucination in Image Captioningcited by 669
  5. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  13. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  14. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  15. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  16. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  17. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  18. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  19. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  20. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  21. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  22. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  23. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  24. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  25. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  26. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  27. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  28. 2014Making Risk Minimization Tolerant to Label Noisecited by 54
  29. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  30. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  31. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  32. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 35
  33. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  34. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Modelcited by 26
  35. 2022On Modality Bias Recognition and Reductioncited by 20
  36. 2021Deconfounded Visual Groundingcited by 16
  37. 2024Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playingcited by 9
  38. 2024Optimal Transport Guided Correlation Assignment for Multimodal Entity Linkingcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  2. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 9
  3. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  4. 2025Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observationscited by 4
  5. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  6. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  7. 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMscited by 2
  8. 2026Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMscited by 2
  9. 2025LaViPlan : Language-Guided Visual Path Planning with RLVRcited by 1
  10. 2025Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge Retrievalcited by 1
  11. 2025QIRL: Optimized Question-Image Relation Learning for Bias-Robust Visual Question Answeringcited by 0
  12. 2025Thought-For-Food: Reasoning Chain Induced Food Visual Question Answeringcited by 0
2 more
  1. 2026EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supportscited by 0
  2. 2026Sensory Precision Inference for Multimodal Arbitration under Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
  2. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
  3. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignment
  4. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  5. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining
  6. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
  7. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  8. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models