mlsanityData to 7 Oct 2026

Measuring Agreeableness Bias in Multimodal Models

arXiv 2408.09111 ↗First posted 17 Aug 2024cs.AI cs.CL cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper examines a phenomenon in multimodal language models where pre-marked options in question images can significantly influence model responses. Our study employs a systematic methodology to investigate this effect: we present models with images of multiple-choice questions, which they initially answer correctly, then expose the same model to versions with pre-marked options. Our findings reveal a significant shift in the models' responses towards the pre-marked option, even when it contradicts their answers in the neutral settings. Comprehensive evaluations demonstrate that this agreeableness bias is a consistent and quantifiable behavior across various model architectures. These results show potential limitations in the reliability of these models when processing images with pre-marked options, raising important questions about their application in critical decision-making contexts where such visual cues might be present.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 22 references that are in the map. First posted on arXiv 17 Aug 2024.

Builds on

17 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study2 Jul
  2. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  3. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?14 Jun
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  5. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  7. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  8. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  9. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  10. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  11. 2024BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Modelscited by 17
  12. 2022Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attentioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  2. 2026Moral Sycophancy in Vision Language Modelscited by 3
  3. 2026GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Themcited by 0
  4. 2026T-SMART: Mechanism-Level Attribution for Tool-Augmented Time-Series Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA
  2. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs
  3. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
  4. 2025Measuring Epistemic Humility in Multimodal Large Language Models
  5. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions
  6. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
  7. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
  8. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict