# Exploring Task-Level Optimal Prompts for Visual In-Context Learning

arXiv: https://arxiv.org/abs/2501.08841 · HTML: https://mlsanity.com/paper/2501.08841/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2025.

## Built on by (most cited first, 4 of 4)

- [Learning to Select Visual In-Context Demonstrations](https://mlsanity.com/paper/2603.26775/) (2026)
- [Enhancing Visual In-Context Learning by Multi-Faceted Fusion](https://mlsanity.com/paper/2601.10107/) (2026)
- [Beyond Single Prompts: Synergistic Fusion and Arrangement for VICL](https://mlsanity.com/paper/2601.10117/) (2026)
- [Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning](https://mlsanity.com/paper/2604.03657/) (2026)

## Abstract (arXiv)

With the development of Vision Foundation Models (VFMs) in recent years, Visual In-Context Learning (VICL) has become a better choice compared to modifying models in most scenarios. Different from retraining or fine-tuning model, VICL does not require modifications to the model's weights or architecture, and only needs a prompt with demonstrations to teach VFM how to solve tasks. Currently, significant computational cost for finding optimal prompts for every test sample hinders the deployment of VICL, as determining which demonstrations to use for constructing prompts is very costly. In this paper, however, we find a counterintuitive phenomenon that most test samples actually achieve optimal performance under the same prompts, and searching for sample-level prompts only costs more time but results in completely identical prompts. Therefore, we propose task-level prompting to reduce the cost of searching for prompts during the inference stage and introduce two time-saving yet effective task-level prompt search strategies. Extensive experimental results show that our proposed method can identify near-optimal prompts and reach the best VICL performance with a minimal cost that prior work has never achieved.