mlsanityData to 7 Oct 2026

SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

arXiv 2606.20244 ↗First posted 18 Jun 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: https://github.com/YinBo0927/SPOT-E

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 55 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

50 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models14 Nov
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2025Qwen3-VL Technical Reportcited by 2,534
36 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  13. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  15. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  16. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  17. 2022Visual Prompting via Image Inpaintingcited by 248
  18. 2023Do Large Language Models Know What They Don't Know?cited by 238
  19. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  20. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  21. 2023Fine-Grained Visual Promptingcited by 81
  22. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 40
  23. 2025Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scalingcited by 30
  24. 2020Overinterpretation reveals image classification model pathologiescited by 28
  25. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  26. 2025Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Modelscited by 16
  27. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  28. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  29. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  30. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  31. 2025FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuningcited by 7
  32. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 6
  33. 2025Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Modelscited by 5
  34. 2025Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuningcited by 4
  35. 2025Boosting Reasoning in Large Multimodal Models via Activation Replaycited by 4
  36. 2026Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behaviorcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
  2. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
  3. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  4. 2026Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence
  5. 2026Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models
  6. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
  7. 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning
  8. 2026Test-Time Scaling for Small VLMs on Multilingual Visual MCQ