mlsanityData to 7 Oct 2026

MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models

arXiv 2609.33431 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: MLLM training and visual instruction tuning

Abstract (arXiv)

We release MoGround, a vision-language dataset spanning four visual domains in which the answer to every question is guaranteed to be available from exactly one modality. This guarantee enables us to measure modality distraction, the failure in which a model answers a question correctly from one modality alone and then flips to a wrong answer once irrelevant content from the other modality is added. Existing probes rarely establish single-modality answerability this way, making it hard to isolate distraction in the first place. Across seven open-source VLMs, we find that modality distraction is not universal but model-dependent. The weaker-grounded modality is the more distracted one (r = +0.86), and distraction scales inversely with grounding strength (r = -0.90). The single-modality guarantee also enables a mitigation method that needs to distinguish between relevant and irrelevant context. Trained on one split of MoGround alone, a weight-space robustness vector reduces distraction on all seven models by 9% to 51%, at a cost of only 0.1 average points of accuracy on standard multimodal tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 39 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

33 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark9 Jun
  2. 2026Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?8 Jun
  3. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models18 Jul
  4. 2024Probing Visual Language Priors in VLMs31 Dec
  5. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  7. 2023Revisiting the Role of Language Priors in Vision-Language Models2 Jun

Other references · 26

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
14 more
  1. 2024Scaling and evaluating sparse autoencoderscited by 530
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  3. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  4. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  5. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  6. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  7. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  8. 2018How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrievalcited by 73
  9. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  10. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  11. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  12. 2025CaTS-Bench: Can Language Models Describe Time Series?cited by 7
  13. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  14. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On the robustness of multimodal language model towards distractions
  2. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models
  3. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
  4. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
  5. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
  6. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
  7. 2025A Video Is Not Worth a Thousand Words
  8. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models