mlsanityData to 7 Oct 2026

LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

arXiv 2608.11847 ↗First posted 12 Aug 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-N selection over existing baselines with negligible additional overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 42 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

27 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation12 Jan

Other references · 26

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
14 more
  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  2. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  3. 2025Learning to Reason without External Rewardscited by 199
  4. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  5. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  6. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  7. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  8. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  9. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 21
  10. 2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
  11. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  12. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  13. 2025Training-free LLM Verification via Recycling Few-shot Examplescited by 4
  14. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decodingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  2. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
  3. 2025When to Think and When to Look: Uncertainty-Guided Lookback
  4. 2025Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
  5. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models
  6. 2026Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
  7. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  8. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks