mlsanityData to 7 Oct 2026

Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

arXiv 2501.01377 ↗First posted 2 Jan 2025cs.CL cs.AI cs.CVcited by 42

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Existing Medical Large Vision-Language Models (Med-LVLMs), encapsulating extensive medical knowledge, demonstrate excellent capabilities in understanding medical images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed to unveil medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward. Experimental results demonstrate that our UMed-LVLM significantly outperforms existing Med-LVLMs in identifying and understanding medical abnormalities, achieving a 58% improvement over the baseline. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 67 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

44 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization9 Dec
  2. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  3. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare27 Oct
  4. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue21 Jun
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  6. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study30 Sep

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2020Learning to summarize from human feedbackcited by 2,562
26 more
  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  12. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  13. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  14. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  15. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  16. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  17. 2020MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysiscited by 206
  18. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  19. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  20. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 85
  21. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  22. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  23. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  24. 2017DeepLesion: Automated Deep Mining, Categorization and Detection of Significant Radiology Image Findings using Large-Scale Clinical Lesion Annotationscited by 23
  25. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  26. 2023Reinforcement Learning for Generative AI: A Surveycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 14.

42 papers.

Sort:

By area:

  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  2. 2025An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editingcited by 6
  3. 2025Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMscited by 4
  4. 2025Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMscited by 3
  5. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Followingcited by 3
  6. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3
  7. 2026Compatibility-Aware Dynamic Fine-Tuning for Large Language Modelscited by 3
  8. 2025Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Modelscited by 2
  9. 2025MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answeringcited by 2
  10. 2025Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Maskscited by 2
  11. 2025Evidence-Driven Marker Extraction for Social Media Suicide Risk Detectioncited by 1
  12. 2025Generating Medically-Informed Explanations for Depression Detection using LLMscited by 1
30 more
  1. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMscited by 1
  2. 2025CAP-LLM: Context-Augmented Personalized Large Language Models for News Headline Generationcited by 1
  3. 2025LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generationcited by 1
  4. 2025DocRefine: An Intelligent Framework for Scientific Document Understanding and Content Optimization based on Multimodal Large Model Agentscited by 1
  5. 2025VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videoscited by 1
  6. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Groundingcited by 1
  7. 2025Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domainscited by 1
  8. 2025Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidancecited by 1
  9. 2025Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sourcescited by 0
  10. 2025Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performancecited by 0
  11. 2025Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detectioncited by 0
  12. 2025Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generationcited by 0
  13. 2025ContextGuard-LVLM: Enhancing News Veracity through Fine-grained Cross-modal Contextual Consistency Verificationcited by 0
  14. 2025Multi-Level LVLM Guidance for Untrimmed Video Action Recognitioncited by 0
  15. 2025Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenizationcited by 0
  16. 2025Enhancing Factual Accuracy and Citation Generation in LLMs via Multi-Stage Self-Verificationcited by 0
  17. 2025Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localizationcited by 0
  18. 2025Context-Enhanced Granular Edit Representation for Efficient and Accurate ASR Post-editingcited by 0
  19. 2026Contextual Discrepancy-Aware Contrastive Learning for Robust Medical Time Series Diagnosis in Small-Sample Scenarioscited by 0
  20. 2026AE-LLM: Adaptive Efficiency Optimization for Large Language Modelscited by 0
  21. 2026Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMscited by 0
  22. 2026EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Modelscited by 0
  23. 2026HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classificationcited by 0
  24. 2026MIThinker: A Plug-and-Play Policy-Optimized Thinker For Motivational Interviewing Counselingcited by 0
  25. 2026BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applicationscited by 0
  26. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  27. 2026Multi-Image Visual Token Pruning in Large Visual Language Modelscited by 0
  28. 2026Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understandingcited by 0
  29. 2026From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Healthcited by 0
  30. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions
  2. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering
  3. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
  4. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  5. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  6. 2025Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models
  7. 2025Medical Large Vision Language Models with Multi-Image Visual Ability
  8. 2025MedM-VL: What Makes a Good Medical LVLM?