mlsanityData to 14 Oct 2026

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

arXiv 2608.17205 ↗First posted 17 Aug 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model's preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at https://github.com/Ro-netizen004/multimodal-arbitration-artifact.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 35 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

19 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap17 Apr
  2. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts10 Apr
  3. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?4 Feb
  4. 2025GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts29 Sep
  5. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models18 Jul
  6. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?2 Jul
  7. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar

Other references · 12

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  12. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  3. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
  4. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  5. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
  6. 2025Test-Time Consistency in Vision Language Models
  7. 2025SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
  8. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread