mlsanityData to 7 Oct 2026

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

arXiv 2609.20139 ↗First posted 17 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust---e.g., rephrasing "Is there a cat?" into "Please look carefully and answer: is there a cat?". Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., "what colour is the cup left of the chair?" instead of "is there a cup?". Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70--81% on the 8B models. The practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 40 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

30 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Same Answer, Different Representations: Hidden instability in VLMs6 Feb
  2. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions18 Apr
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 27

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  8. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  9. 2015Neural Module Networkscited by 375
  10. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  11. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  12. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
15 more
  1. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  2. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  3. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  4. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  5. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  6. 2016Understanding How Image Quality Affects Deep Neural Networkscited by 118
  7. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  8. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 97
  9. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  10. 2021Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domaincited by 70
  11. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  12. 2023Localizing and Editing Knowledge in Text-to-Image Generative Modelscited by 52
  13. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  14. 2018On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functionscited by 31
  15. 2020Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustnesscited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Helpful Text Hurts: Option-Redirecting Bias in Vision-Language Models
  2. 2025Questioning the Stability of Visual Question Answering
  3. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
  4. 2026Diagnosing Visual Ignorance in Vision-Language Models
  5. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  6. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  7. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
  8. 2026Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models