mlsanityData to 14 Oct 2026

Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models

arXiv 2505.16647 ↗First posted 22 May 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We investigate fine-tuning Vision-Language Models (VLMs) for multi-task medical image understanding, focusing on detection, localization, and counting of findings in medical images. Our objective is to evaluate whether instruction-tuned VLMs can simultaneously improve these tasks, with the goal of enhancing diagnostic accuracy and efficiency. Using MedMultiPoints, a multimodal dataset with annotations from endoscopy (polyps and instruments) and microscopy (sperm cells), we reformulate each task into instruction-based prompts suitable for vision-language reasoning. We fine-tune Qwen2.5-VL-7B-Instruct using Low-Rank Adaptation (LoRA) across multiple task combinations. Results show that multi-task training improves robustness and accuracy. For example, it reduces the Count Mean Absolute Error (MAE) and increases Matching Accuracy in the Counting + Pointing task. However, trade-offs emerge, such as more zero-case point predictions, indicating reduced reliability in edge cases despite overall performance gains. Our study highlights the potential of adapting general-purpose VLMs to specialized medical tasks via prompt-driven fine-tuning. This approach mirrors clinical workflows, where radiologists simultaneously localize, count, and describe findings - demonstrating how VLMs can learn composite diagnostic reasoning patterns. The model produces interpretable, structured outputs, offering a promising step toward explainable and versatile medical AI. Code, model weights, and scripts will be released for reproducibility at https://github.com/simula/PointDetectCount.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 33 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

20 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation5 May
  2. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Models20 Mar
  3. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations4 Jan
  4. 2024MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation29 Sep
  5. 2024LiteGPT: Large Vision-Language Model for Joint Chest X-ray Localization and Classification Task16 Jul

Other references · 15

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2017A Survey on Multi-Task Learningcited by 840
  6. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 589
  7. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  8. 2023Towards Generalist Biomedical AIcited by 289
  9. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  10. 2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87
  11. 2023CLIP in Medical Imaging: A Surveycited by 40
  12. 2019A Review of Object Detection Models based on Convolutional Neural Networkcited by 22
3 more
  1. 2023DeGPR: Deep Guided Posterior Regularization for Multi-Class Cell Detection and Countingcited by 18
  2. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuningcited by 5
  3. 2022VISEM-Tracking, a human spermatozoa tracking datasetcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  2. 2026Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
  2. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning
  3. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classification
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  6. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  7. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  8. 2025Test-Time-Scaling for Zero-Shot Diagnosis with Visual-Language Reasoning