Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace this paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show that question-first prompting steers perception, shifting image patch representations toward question-relevant concepts. But downstream, stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven, often wrong answers. Causal attention knockout confirms that the answer reads the question only when it follows the image. This diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so one copy steers perception while the other is available at answer time. A similar division of labor appears in a fifty-year-old finding on human 'adjunct questions', where repeating a question before and after a passage improves comprehension. Echoing the image as well brings further gains by restoring the whole-image view otherwise lost by a causal decoder. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts recover most of the gap and, on NaturalBench and Winoground, surpass the best single-pass ordering by up to 19 group-accuracy points on Winoground, with no training, fine-tuning, or architecture change. The paradox reveals a tension between steering what a model sees and preserving access to what it was asked; echoing resolves this through prompt design. Project Page: https://rakshanda-cmu.github.io/ask-twice-look-twice/
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 23 references that are in the map. First posted on arXiv 17 Jul 2026.
Builds on
10 of 23 references.
Closest ancestors · 2
Same direction, newest first
- 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct
- 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
Other references · 8
- 2023Visual Instruction Tuningcited by 7,792
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
- 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 19
- 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
- 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Readingcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
- 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
- 2026Mirage Probes: How Vision Models Fake Visual Understanding
- 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
- 2025Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering
- 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions
- 2026Seeing and Solving Are Not Enough for Vision-Language Models
- 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models