mlsanityData to 7 Oct 2026

A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models

arXiv 2605.20495 ↗First posted 19 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Deep-learning pipelines for microscopy image classification often require expensive, labor- and time-intensive expert annotation to produce high-quality ground truth for training. Recent work has shown that prompt tuning of vision-language models (VLMs) can reduce manual annotation by constructing a small prompt set of expert-verified image-caption exemplars that is reused as few-shot context to classify all remaining images at inference time. To further reduce effort, the VLM can draft captions for candidate exemplars, which experts then verify and lightly edit instead of writing text de novo. However, two practical questions remain unaddressed: (1) which unlabeled images should be prioritized for verification, and (2) how many verified exemplars are needed to reach a performance target. In this work, we address these questions by formulating prompt-set construction as a target-driven active learning problem that prioritizes which images to annotate. We study three complementary selection criteria under strict low-resource constraints with small unlabeled pools. Experiments show that our methods reach the target performance with substantially fewer expert-verified images than random selection, achieving 100% test accuracy with as few as 20 annotated images on average. More broadly, our human-in-the-loop framework demonstrates a human-centered use of generative AI in biomedical image analysis, where experts remain actively involved in verifying and refining model output while significantly reducing annotation cost. Code and data will be publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 33 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

19 of 33 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  8. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  9. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  10. 2025MedGemma Technical Reportcited by 398
  11. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
7 more
  1. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  2. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  3. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  4. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  5. 2023Active Prompt Learning in Vision Language Modelscited by 25
  6. 2024Active Learning for Vision-Language Modelscited by 15
  7. 2024Active Prompt Tuning Enables Gpt-40 To Do Efficient Classification Of Microscopy Imagescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models
  2. 2025In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy
  3. 2024Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis
  4. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
  5. 2025Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
  6. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  7. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  8. 2026Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models