mlsanityData to 14 Oct 2026

Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learning

arXiv 2604.03657 ↗First posted 4 Apr 2026cs.CV cs.IR cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 4 Apr 2026.

Builds on

38 of 46 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  12. 2022Transformers learn in-context by gradient descentcited by 732
26 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  2. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  3. 2021MetaICL: Learning to Learn In Contextcited by 454
  4. 2017One-Shot Learning for Semantic Segmentationcited by 337
  5. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  6. 2022Visual Prompting via Image Inpaintingcited by 250
  7. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  8. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  9. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  10. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  11. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  12. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  13. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 58
  14. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  15. 2023Explore In-Context Learning for 3D Point Cloud Understandingcited by 40
  16. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  17. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  18. 2024Towards Global Optimal Visual In-Context Learning Prompt Selectioncited by 15
  19. 2025Stable Diffusion Models are Secretly Good at Visual In-Context Learningcited by 8
  20. 2024DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Expertscited by 7
  21. 2025Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learningcited by 7
  22. 2024Mixtures of In-Context Learnerscited by 5
  23. 2025Exploring Task-Level Optimal Prompts for Visual In-Context Learningcited by 4
  24. 2026Learning Cross-View Object Correspondence via Cycle-Consistent Mask Predictioncited by 3
  25. 2025Towards Reliable and Holistic Visual In-Context Learning Prompt Selectioncited by 2
  26. 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
  2. 2024Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations
  3. 2025AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs
  4. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  5. 2026Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
  6. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model
  7. 2024TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt
  8. 2026GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models