# Towards Global Optimal Visual In-Context Learning Prompt Selection

arXiv: https://arxiv.org/abs/2405.15279 · HTML: https://mlsanity.com/paper/2405.15279/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [What Makes Good Examples for Visual In-Context Learning?](https://mlsanity.com/paper/2301.13670/) (2023)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [In-Context Learning Unlocked for Diffusion Models](https://mlsanity.com/paper/2305.01115/) (2023)
- [Instance Credibility Inference for Few-Shot Learning](https://mlsanity.com/paper/2003.11853/) (2020)
- [Exploring Effective Factors for Improving Visual In-Context Learning](https://mlsanity.com/paper/2304.04748/) (2023)
- [NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sorting](https://mlsanity.com/paper/2102.07831/) (2021)
- [Explore In-Context Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2306.08659/) (2023)
- [RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank](https://mlsanity.com/paper/2305.16726/) (2023)
- [Deep Attentive Ranking Networks for Learning to Order Sentences](https://mlsanity.com/paper/2001.00056/) (2019)
- [How to trust unlabeled data? Instance Credibility Inference for Few-Shot Learning](https://mlsanity.com/paper/2007.08461/) (2020)
- [Scalable Penalized Regression for Noise Detection in Learning with Noisy Labels](https://mlsanity.com/paper/2203.07788/) (2022)
- [HodgeRank with Information Maximization for Crowdsourced Pairwise Ranking Aggregation](https://mlsanity.com/paper/1711.05957/) (2017)
- [Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels](https://mlsanity.com/paper/2301.00545/) (2023)

## Built on by (most cited first, 15 of 15)

- [Visual Prompting in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2409.15310/) (2024)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning](https://mlsanity.com/paper/2504.21263/) (2025)
- [PAVE: Patching and Adapting Video Large Language Models](https://mlsanity.com/paper/2503.19794/) (2025)
- [Towards Reliable and Holistic Visual In-Context Learning Prompt Selection](https://mlsanity.com/paper/2509.25989/) (2025)
- [PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment](https://mlsanity.com/paper/2603.18891/) (2026)
- [E-InMeMo: Enhanced Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2504.18158/) (2025)
- [PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learning](https://mlsanity.com/paper/2509.21926/) (2025)
- [Enhancing Visual In-Context Learning by Multi-Faceted Fusion](https://mlsanity.com/paper/2601.10107/) (2026)
- [Cycle Context Verification for In-Context Medical Image Segmentation](https://mlsanity.com/paper/2507.08357/) (2025)
- [FlexICL: A Flexible Visual In-context Learning Framework for Elbow and Wrist Ultrasound Segmentation](https://mlsanity.com/paper/2510.26049/) (2025)
- [Beyond Single Prompts: Synergistic Fusion and Arrangement for VICL](https://mlsanity.com/paper/2601.10117/) (2026)
- [Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning](https://mlsanity.com/paper/2604.03657/) (2026)
- [Toward Robust In-Context Segmentation via Concept Guidance](https://mlsanity.com/paper/2606.28149/) (2026)
- [PromptPath: Prompt-Adaptive Computational Pathways for In-Context Learning](https://mlsanity.com/paper/2608.02129/) (2026)

## Abstract (arXiv)

Visual In-Context Learning (VICL) is a prevailing way to transfer visual foundation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query sample. The fundamental problem in VICL is how to select the best prompt to activate its power as much as possible, which is equivalent to the ranking problem to test the in-context behavior of each candidate in the alternative set and select the best one. To utilize more appropriate ranking metric and leverage more comprehensive information among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.e. choosing the best performing in-context examples from all alternatives for each query sample. Our method, dubbed Partial2Global, adopts a transformer-based list-wise ranker to provide a more comprehensive comparison within several alternatives, and a consistency-aware ranking aggregator to generate globally consistent ranking. The effectiveness of Partial2Global is validated through experiments on foreground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establish the new state-of-the-arts.