mlsanityData to 7 Oct 2026

BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA

arXiv 2603.28026 ↗First posted 30 Mar 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal multiple-choice question answering (MCQA) provides a standardized and objectively measurable setting for evaluating vision-language models (VLMs). However, because the MCQA format incorporates the candidate choices into the input context, it introduces several unintended biases. Previous work has primarily focused on structural biases, such as preferences for certain choices. Instead, we argue that the choices act as textual priors, causing models to favor linguistically plausible options regardless of the visual content. We hypothesize and empirically verify that a model genuinely relies on visual evidence only when its multimodal distribution significantly diverges from its text-only distribution. Based on this observation we propose BUZZY,a training-free decoding method that corrects multimodal predictions by subtracting the text-only distribution. Experiments with five VLMs on five multimodal MCQA benchmarks demonstrate that BUZZY achieves the highest average accuracy among state-of-the-art methods while reducing inference latency by over 28% compared to prior contrastive decoding approaches. Overall, these results suggest that amplifying the visual signal by penalizing text-only preferences is key to efficient and robust multimodal MCQA reasoning. Code and additional resources are provided https://txxnrd.github.io/buzzy/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 34 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

27 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding14 Aug
  2. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation14 May
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 23

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  8. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  10. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  11. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  12. 2024Probing Visual Language Priors in VLMscited by 44
11 more
  1. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  2. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 28
  3. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  4. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  5. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
  6. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  7. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Modelscited by 20
  8. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Modelscited by 12
  9. 2024Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Modelscited by 11
  10. 2023Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyondcited by 7
  11. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective
  2. 2024Measuring Agreeableness Bias in Multimodal Models
  3. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors
  4. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
  5. 2025Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
  6. 2025Evaluating Variance in Visual Question Answering Benchmarks
  7. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
  8. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation