# Measuring Agreeableness Bias in Multimodal Models

arXiv: https://arxiv.org/abs/2408.09111 · HTML: https://mlsanity.com/paper/2408.09111/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 17 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://mlsanity.com/paper/2405.05949/) (2024)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective](https://mlsanity.com/paper/2403.18346/) (2024)
- [CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?](https://mlsanity.com/paper/2403.04547/) (2024)
- [MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs](https://mlsanity.com/paper/2406.17126/) (2024)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Understanding Alignment in Multimodal LLMs: A Comprehensive Study](https://mlsanity.com/paper/2407.02477/) (2024)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models](https://mlsanity.com/paper/2407.15240/) (2024)
- [Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention](https://mlsanity.com/paper/2211.11701/) (2022)

## Built on by (most cited first, 4 of 4)

- [Sycophancy Claims about Language Models: The Missing Human-in-the-Loop](https://mlsanity.com/paper/2512.00656/) (2025)
- [Moral Sycophancy in Vision Language Models](https://mlsanity.com/paper/2602.08311/) (2026)
- [GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them](https://mlsanity.com/paper/2609.06058/) (2026)
- [T-SMART: Mechanism-Level Attribution for Tool-Augmented Time-Series Question Answering](https://mlsanity.com/paper/2609.14142/) (2026)

## Abstract (arXiv)

This paper examines a phenomenon in multimodal language models where pre-marked options in question images can significantly influence model responses. Our study employs a systematic methodology to investigate this effect: we present models with images of multiple-choice questions, which they initially answer correctly, then expose the same model to versions with pre-marked options. Our findings reveal a significant shift in the models' responses towards the pre-marked option, even when it contradicts their answers in the neutral settings. Comprehensive evaluations demonstrate that this agreeableness bias is a consistent and quantifiable behavior across various model architectures. These results show potential limitations in the reliability of these models when processing images with pre-marked options, raising important questions about their application in critical decision-making contexts where such visual cues might be present.