# Tinted Frames: Question Framing Blinds Vision-Language Models

arXiv: https://arxiv.org/abs/2603.19203 · HTML: https://mlsanity.com/paper/2603.19203/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training](https://mlsanity.com/paper/2509.23661/) (2025)
- [See What You Are Told: Visual Attention Sink in Large Multimodal Models](https://mlsanity.com/paper/2503.03321/) (2025)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [Revisiting the Role of Language Priors in Vision-Language Models](https://mlsanity.com/paper/2306.01879/) (2023)
- [From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks](https://mlsanity.com/paper/2406.06579/) (2024)
- [To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models](https://mlsanity.com/paper/2510.08510/) (2025)
- [Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models](https://mlsanity.com/paper/2505.16416/) (2025)
- [A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models](https://mlsanity.com/paper/2307.12980/) (2023)
- [Hidden in plain sight: VLMs overlook their visual representations](https://mlsanity.com/paper/2506.08008/) (2025)
- [LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision](https://mlsanity.com/paper/2505.18051/) (2025)
- [Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models](https://mlsanity.com/paper/2509.21984/) (2025)

## Abstract (arXiv)

Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.