mlsanityData to 14 Oct 2026

Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy

arXiv 2609.00067 ↗First posted 30 Aug 2026cs.CL cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

External text can override conflicting image evidence in multimodal large language models, a failure we call multimodal contextual sycophancy. We introduce a 998-case diagnostic that independently varies visual evidence, commonsense priors, and external text, and probe when this failure arises by moving the information boundary around a context-blind visual witness. On abnormal images paired with Gemini-generated false text, GPT-5.1 scores 7.9% under joint conditioning, 49.7% when the context-blind witness report is scored directly, 63.7% under a matched two-call witness-arbiter pipeline that exposes the witness to the text, and 84.2% under System-2 Visual Arbitration (S2VA), which withholds the text from the witness. Across six models, S2VA improves over the direct witness report by 19.7 to 44.1 points, with all paired 95% confidence intervals excluding zero. The best information boundary is not uniform: textual context scaffolds some models, and a GPT-4o-regenerated subset changes the relative ordering of joint conditioning, Witness-Only, and S2VA. Contextual sycophancy is therefore sensitive to when text is introduced, as well as to the model and context source.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 18 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

7 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs10 Oct

Other references · 6

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  4. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  5. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  6. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  2. 2026Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure
  3. 2026CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
  4. 2026When Prompts Mislead: Textual Dominance and Diagnostic Bias in MLLMs
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  6. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models
  7. 2026Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment
  8. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability