mlsanityData to 7 Oct 2026

Biomedical Visual Instruction Tuning with Clinician Preference Alignment

arXiv 2406.13173 ↗First posted 19 Jun 2024cs.CV cs.AI cs.CLcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

Abstract (arXiv)

Recent advancements in multimodal foundation models have showcased impressive capabilities in understanding and reasoning with visual and textual information. Adapting these foundation models trained for general usage to specialized domains like biomedicine requires large-scale domain-specific instruction datasets. While existing works have explored curating such datasets automatically, the resultant datasets are not explicitly aligned with domain expertise. In this work, we propose a data-centric framework, Biomedical Visual Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that incorporates clinician preferences into both stages of generating and selecting instruction data for tuning biomedical multimodal foundation models. First, during the generation stage, we prompt the GPT-4V generator with a diverse set of clinician-selected demonstrations for preference-aligned data candidate generation. Then, during the selection phase, we train a separate selection model, which explicitly distills clinician and policy-guided model preferences into a rating function to select high-quality data for medical instruction tuning. Results show that the model tuned with the instruction-following data from our method demonstrates a significant improvement in open visual chat (18.5% relatively) and medical VQA (win rate up to 81.73%). Our instruction-following data and models are available at BioMed-VITAL.github.io.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 50 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

40 of 50 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
28 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  9. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  13. 2023Towards Generalist Biomedical AIcited by 286
  14. 2024Capabilities of Gemini Models in Medicinecited by 231
  15. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  16. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  17. 2023Text Classification via Large Language Modelscited by 104
  18. 2023GPT-RE: In-context Learning for Relation Extraction using Large Language Modelscited by 62
  19. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  20. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  21. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 54
  22. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  23. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  24. 2022Self-supervised vision-language pretraining for Medical visual question answeringcited by 30
  25. 2023Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answeringcited by 29
  26. 2023Large language models in medicine: the potentials and pitfallscited by 21
  27. 2023Album Storytelling with Iterative Story-aware Captioning and Large Language Modelscited by 7
  28. 2024Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Modelcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 6.

21 papers.

Sort:

By area:

  1. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  2. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  3. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  4. 2025CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Modelscited by 13
  5. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  6. 2025Uncertainty-Aware Large Language Models for Explainable Disease Diagnosiscited by 9
  7. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learningcited by 6
  8. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  9. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  10. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  11. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  12. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicinecited by 1
9 more
  1. 2026Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Taskscited by 1
  2. 2024MGH Radiology Llama: A Llama 3 70B Model for Radiologycited by 0
  3. 2025AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answeringcited by 0
  4. 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoningcited by 0
  5. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Modelscited by 0
  6. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0
  7. 2026Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Modelscited by 0
  8. 2026Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncologycited by 0
  9. 2026DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding
  2. 2025How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?
  3. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  4. 2025Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications
  5. 2025IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment
  6. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Application
  7. 2025Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models
  8. 2026Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation