mlsanityData to 7 Oct 2026

STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering

arXiv 2406.19973 ↗First posted 28 Jun 2024cs.CV cs.LGcited by 21

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 2 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have shown significant potential in assisting medical diagnosis by leveraging extensive biomedical datasets. However, the advancement of medical image understanding and reasoning critically depends on building high-quality visual instruction data, which is costly and labor-intensive to obtain, particularly in the medical domain. To mitigate this data-starving issue, we introduce Self-Training Large Language and Vision Assistant for Medicine (STLLaVA-Med). The proposed method is designed to train a policy model (an LVLM) capable of auto-generating medical visual instruction data to improve data efficiency, guided through Direct Preference Optimization (DPO). Specifically, a more powerful and larger LVLM (e.g., GPT-4o) is involved as a biomedical expert to oversee the DPO fine-tuning process on the auto-generated data, encouraging the policy model to align efficiently with human preferences. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks, demonstrating competitive zero-shot performance with the utilization of only 9% of the medical data.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

21 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
7 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  2. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  5. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  6. 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
  7. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistantcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systemscited by 26
  2. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Modelcited by 19
  3. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  4. 2025MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progressioncited by 9
  5. 2025MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Modelscited by 6
  6. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  7. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  8. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  9. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 3
  10. 2025Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Modelscited by 2
  11. 2025X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoningcited by 2
  12. 2025Visual Self-Refinement for Autoregressive Modelscited by 2
9 more
  1. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  2. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1
  3. 2025Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answeringcited by 1
  4. 2026ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Modelscited by 1
  5. 2025Q-FSRU: Quantum-Augmented Frequency-Spectral Fusion for Medical Visual Question Answeringcited by 0
  6. 2025Q-FSRU: Quantum-Augmented Frequency-Spectral For Medical Visual Question Answeringcited by 0
  7. 2026Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMscited by 0
  8. 2026Information-Regularized Attention for Visual-Centric Reasoningcited by 0
  9. 2026Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
  2. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
  3. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  4. 2025CAPO: Reinforcing Consistent Reasoning in Medical Decision-Making
  5. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
  6. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
  7. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  8. 2022Self-supervised vision-language pretraining for Medical visual question answering