mlsanityData to 7 Oct 2026

Learning What Helps: Task-Aligned Context Selection for Vision Tasks

arXiv 2512.00489 ↗First posted 29 Nov 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans often resolve visual uncertainty by comparing an image with relevant examples, but ViTs lack the ability to identify which examples would improve their predictions. We present Task-Aligned Context Selection (TACS), a framework that learns to select paired examples which truly improve task performance rather than those that merely appear similar. TACS jointly trains a selector network with the task model through a hybrid optimization scheme combining gradient-based supervision and reinforcement learning, making retrieval part of the learning objective. By aligning selection with task rewards, TACS enables discriminative models to discover which contextual examples genuinely help. Across 18 datasets covering fine-grained recognition, medical image classification, and medical image segmentation, TACS consistently outperforms similarity-based retrieval, particularly in challenging or data-limited settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 29 Nov 2025.

Builds on

31 of 49 references.

References · 31

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2021Vision Transformers for Dense Predictioncited by 1,789
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
  12. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
19 more
  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  2. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  3. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  4. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 224
  5. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  6. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  7. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  8. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  9. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  10. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  11. 2023Are Natural Domain Foundation Models Useful for Medical Image Classification?cited by 44
  12. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  13. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  14. 2024Retrieval-augmented Few-shot Medical Image Segmentation with Foundation Modelscited by 20
  15. 2021Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimationcited by 19
  16. 2024SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedbackcited by 19
  17. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  18. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Modelscited by 9
  19. 2025RAU: Reference-based Anatomical Understanding with Vision Language Modelscited by 3

Built on by

Not cited yet.