mlsanityData to 14 Oct 2026

Test-Time Hinting for Black-Box Vision-Language Models

arXiv 2605.16410 ↗First posted 13 May 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to vision-language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on multimodal mathematical and scientific reasoning benchmarks rather than general visual understanding tasks. In this paper, we propose Test-Time Hinting, a method that improves VLM performance via a single VLM call and requiring only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes. We show that Test-Time Hinting improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks and that these gains generalize to unseen benchmarks and VLMs without retraining the hint generator.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 24 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

19 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning27 Nov
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 15

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
3 more
  1. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  2. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  3. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Test-Time Scaling for Vision-Language Models
  2. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  3. 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoning
  4. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  5. 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Models
  6. 2026Test-Time Scaling for Small VLMs on Multilingual Visual MCQ
  7. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  8. 2026Empirical Recipes for Efficient and Compact Vision-Language Models