# Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models

arXiv: https://arxiv.org/abs/2505.13973 · HTML: https://mlsanity.com/paper/2505.13973/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model](https://mlsanity.com/paper/2503.05132/) (2025)
- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2411.14432/) (2024)
- [R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning](https://mlsanity.com/paper/2503.05379/) (2025)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)

## Built on by (most cited first, 8 of 8)

- [LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding](https://mlsanity.com/paper/2508.01617/) (2025)
- [Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning](https://mlsanity.com/paper/2508.02258/) (2025)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models](https://mlsanity.com/paper/2511.14900/) (2025)
- [CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency](https://mlsanity.com/paper/2512.17213/) (2025)
- [Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients](https://mlsanity.com/paper/2512.23090/) (2025)
- [Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2603.07443/) (2026)
- [Reinforcement Learning for Evidence-Seeking Diagnostic Reasoning with Large Language Models](https://mlsanity.com/paper/2607.02983/) (2026)

## Abstract (arXiv)

Recently, reinforcement learning (RL)-based tuning has shifted the trajectory of Multimodal Large Language Models (MLLMs), particularly following the introduction of Group Relative Policy Optimization (GRPO). However, directly applying it to medical tasks remains challenging for achieving clinically grounded model behavior. Motivated by the need to align model response with clinical expectations, we investigate four critical dimensions that affect the effectiveness of RL-based tuning in medical visual question answering (VQA): base model initialization strategy, the role of medical semantic alignment, the impact of length-based rewards on long-chain reasoning, and the influence of bias. We conduct extensive experiments to analyze these factors for medical MLLMs, providing new insights into how models are domain-specifically fine-tuned. Additionally, our results also demonstrate that GRPO-based RL tuning consistently outperforms standard supervised fine-tuning (SFT) in both accuracy and reasoning quality.