# Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

arXiv: https://arxiv.org/abs/2501.01377 · HTML: https://mlsanity.com/paper/2501.01377/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 44 of its 67 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)

## Built on by (most cited first, 30 of 42)

- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editing](https://mlsanity.com/paper/2508.17435/) (2025)
- [Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMs](https://mlsanity.com/paper/2502.15009/) (2025)
- [Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs](https://mlsanity.com/paper/2503.08857/) (2025)
- [Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models](https://mlsanity.com/paper/2508.02886/) (2025)
- [ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following](https://mlsanity.com/paper/2508.15164/) (2025)
- [Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees](https://mlsanity.com/paper/2603.22355/) (2026)
- [Compatibility-Aware Dynamic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2606.11206/) (2026)
- [MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answering](https://mlsanity.com/paper/2508.07023/) (2025)
- [Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks](https://mlsanity.com/paper/2511.06665/) (2025)
- [Evidence-Driven Marker Extraction for Social Media Suicide Risk Detection](https://mlsanity.com/paper/2502.18823/) (2025)
- [Generating Medically-Informed Explanations for Depression Detection using LLMs](https://mlsanity.com/paper/2503.14671/) (2025)
- [CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs](https://mlsanity.com/paper/2507.22074/) (2025)
- [CAP-LLM: Context-Augmented Personalized Large Language Models for News Headline Generation](https://mlsanity.com/paper/2508.03935/) (2025)
- [LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation](https://mlsanity.com/paper/2508.04732/) (2025)
- [DocRefine: An Intelligent Framework for Scientific Document Understanding and Content Optimization based on Multimodal Large Model Agents](https://mlsanity.com/paper/2508.07021/) (2025)
- [VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos](https://mlsanity.com/paper/2508.15903/) (2025)
- [Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding](https://mlsanity.com/paper/2508.16974/) (2025)
- [Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains](https://mlsanity.com/paper/2508.19357/) (2025)
- [Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance](https://mlsanity.com/paper/2509.05669/) (2025)
- [Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sources](https://mlsanity.com/paper/2502.05944/) (2025)
- [Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performance](https://mlsanity.com/paper/2504.14633/) (2025)
- [Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detection](https://mlsanity.com/paper/2506.01104/) (2025)
- [Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generation](https://mlsanity.com/paper/2506.01118/) (2025)
- [ContextGuard-LVLM: Enhancing News Veracity through Fine-grained Cross-modal Contextual Consistency Verification](https://mlsanity.com/paper/2508.06623/) (2025)
- [Multi-Level LVLM Guidance for Untrimmed Video Action Recognition](https://mlsanity.com/paper/2508.17442/) (2025)
- [Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization](https://mlsanity.com/paper/2509.05695/) (2025)
- [Enhancing Factual Accuracy and Citation Generation in LLMs via Multi-Stage Self-Verification](https://mlsanity.com/paper/2509.05741/) (2025)
- [Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localization](https://mlsanity.com/paper/2509.13474/) (2025)
- [Context-Enhanced Granular Edit Representation for Efficient and Accurate ASR Post-editing](https://mlsanity.com/paper/2509.14263/) (2025)

## Abstract (arXiv)

Existing Medical Large Vision-Language Models (Med-LVLMs), encapsulating extensive medical knowledge, demonstrate excellent capabilities in understanding medical images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed to unveil medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward. Experimental results demonstrate that our UMed-LVLM significantly outperforms existing Med-LVLMs in identifying and understanding medical abnormalities, achieving a 58% improvement over the baseline. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.