mlsanityData to 7 Oct 2026

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

arXiv 2604.01280 ↗First posted 1 Apr 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 52 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

41 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding8 Mar
  2. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  12. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
27 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  7. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  8. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  10. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  11. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  12. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  13. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  14. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  15. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  16. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  17. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  18. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  19. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 49
  20. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  21. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inferencecited by 23
  22. 2025SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidencecited by 18
  23. 2025Vision Function Layer in Multimodal LLMscited by 17
  24. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  25. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  26. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  27. 2026QKVQA: Question-Focused Filtering for Knowledge-based VQAcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
  2. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering
  3. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
  4. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  5. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  6. 2025See the Forest and the Trees: A Synergistic Reasoning Framework for Knowledge-Based Visual Question Answering
  7. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  8. 2026Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models