mlsanityData to 7 Oct 2026

AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs

arXiv 2506.16112 ↗First posted 19 Jun 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Inspired by text prompts in large language models, visual prompts have been explored to enhance the perceptual capabilities of large vision-language models (LVLMs). However, performance tends to saturate under single visual prompt designs, making further prompt engineering increasingly ineffective. To address this limitation, we shift from prompt engineering to prompt retrieval and propose AutoV, a lightweight framework for instance-adaptive visual prompt identification. Given an input image and a textual query, AutoV automatically locates the most suitable visual prompt from a diverse candidate pool. Training such a retrieval framework requires prompt-level supervision, yet prompt quality is inherently ambiguous and difficult to assess reliably, even for humans. To enable automatic supervision, we evaluate visual prompts using a pre-trained LVLM and label them according to their prediction losses. Using the loss-oriented ranking as a robust training signal, AutoV learns to retrieve the query-aware optimal prompt for each instance without manual annotation. Experiments indicate that AutoV enhances the performance of various LVLMs on image understanding, captioning, grounding, and classification tasks. For example, AutoV improves LLaVA-OV by 10.2% on VizWiz and boosts Qwen2.5-VL by 3.8% on MMMU, respectively.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 86 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

66 of 86 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMs13 Apr
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar

Other references · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2023Segment Anythingcited by 8,158
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  14. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  15. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  17. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  20. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  21. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  22. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  23. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  24. 2016Modeling Context in Referring Expressionscited by 1,054
  25. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  26. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  27. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  28. 2023Instruction Tuning with GPT-4cited by 676
  29. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  30. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  31. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  32. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  33. 2024InternLM2 Technical Reportcited by 391
  34. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  35. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  36. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  37. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  38. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  2. 2025ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Bettercited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Prompt Pretraining of Vision-Language Models
  2. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification
  3. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  4. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  5. 2025Modeling Variants of Prompts for Vision-Language Models
  6. 2023LaViP:Language-Grounded Visual Prompts
  7. 2024IPO: Interpretable Prompt Optimization for Vision-Language Models
  8. 2025Modular Prompt Learning Improves Vision-Language Models