QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Clinical decision-making routinely demands reasoning over heterogeneous data, yet existing multimodal language models (MLLMs) remain largely vision-centric and fail to generalize across clinical specialties. To bridge this gap, we introduce QoQ-Med-7B/32B, the first open generalist clinical foundation model that jointly reasons across medical images, time-series signals, and text reports. QoQ-Med is trained with Domain-aware Relative Policy Optimization (DRPO), a novel reinforcement-learning objective that hierarchically scales normalized rewards according to domain rarity and modality difficulty, mitigating performance imbalance caused by skewed clinical data distributions. Trained on 2.61 million instruction tuning pairs spanning 9 clinical domains, we show that DRPO training boosts diagnostic performance by 43% in macro-F1 on average across all visual domains as compared to other critic-free training methods like GRPO. Furthermore, with QoQ-Med trained on intensive segmentation data, it is able to highlight salient regions related to the diagnosis, with an IoU 10x higher than open models while reaching the performance of OpenAI o4-mini. To foster reproducibility and downstream research, we release (i) the full model weights, (ii) the modular training pipeline, and (iii) all intermediate reasoning traces at https://github.com/DDVD233/QoQ_Med.
Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 50 of its 97 references that are in the map. First posted on arXiv 31 May 2025.
Builds on
50 of 97 references.
Closest ancestors · 4
Same direction, newest first
- 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models18 Mar
- 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning27 Feb
- 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning26 Feb
- 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning3 Jun
Other references · 46
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
34 more
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
- 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
- 2022Towards Reasoning in Large Language Models: A Surveycited by 596
- 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
- 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
- 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 239
- 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
- 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
- 2020A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Contextcited by 146
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
- 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
- 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
- 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 122
- 2020PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphonescited by 118
- 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
- 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 103
- 2020COVID-19 Image Data Collectioncited by 101
- 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
- 2022Brain Network Transformercited by 94
- 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
- 2020Interpretability of machine learning based prediction models in healthcarecited by 75
- 2022BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networkscited by 75
- 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
- 2022A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Modelscited by 43
- 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
- 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 39
- 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
- 2023INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosiscited by 27
- 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
- 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
- 2024Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecturecited by 19
- 2021COVIDx-US -- An open-access benchmark dataset of ultrasound imaging data for AI-driven COVID-19 analyticscited by 17
- 2024Reward Centeringcited by 16
- 2025CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Modelscited by 13
Built on by
Citing papers per year: 2025: 9; 2026: 29.
38 papers.
Sort:
By area:
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
- 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
- 2026ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretationcited by 9
- 2026MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamicscited by 7
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoningcited by 6
- 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learningcited by 5
- 2025UniECG: Understanding and Generating ECG in One Unified Modelcited by 4
- 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 4
- 2025Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMscited by 3
- 2026When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gainscited by 3
- 2025Automated Structured Radiology Report Generation with Rich Clinical Contextcited by 2
- 2026Visual Memory Injection Attacks for Multi-Turn Conversationscited by 2
26 more
- 2026CTM-AI: A Blueprint for General AI Inspired by a Model of Consciousnesscited by 2
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
- 2026Scaling Participation in Modular AI Systemscited by 2
- 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 2
- 2025The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Qualitycited by 1
- 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Videocited by 1
- 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videoscited by 1
- 2026A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classificationcited by 1
- 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
- 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generationcited by 1
- 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
- 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoningcited by 0
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
- 2026MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicinecited by 0
- 2026AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoningcited by 0
- 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasoundcited by 0
- 2026Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysiscited by 0
- 2026Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classificationcited by 0
- 2026Information-theoretic Multimodal Representation Learning for Electrocardiogram Signalscited by 0
- 2026CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluationscited by 0
- 2026Learning Where to Look: A Reinforcement Learning Framework for Robust Micro-Ultrasound Prostate Cancer Detectioncited by 0
- 2026MedUPS: Towards Diagnostic Assistance in Uncommon Medical Cases with Large Language Modelscited by 0
- 2026Continuous-Latent Predictive Modeling with Semantic Alignment for EEG-Language Foundation Modelscited by 0
- 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0
- 2026TRACE: Expert-Aligned ECG Representation Learning with Rigorous Benchmarking and Real-World Validation in Acute Cardiac Carecited by 0
- 2026Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillationcited by 0
- 2026Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation28 Sep 2026
- 2026TRACE: Expert-Aligned ECG Representation Learning with Rigorous Benchmarking and Real-World Validation in Acute Cardiac Care28 Sep 2026
- 2026Volumetric Radiology AI in the Era of Multimodal Large Language Models20 Aug 2026
- 2026Continuous-Latent Predictive Modeling with Semantic Alignment for EEG-Language Foundation Models12 Aug 2026
- 2026MedUPS: Towards Diagnostic Assistance in Uncommon Medical Cases with Large Language Models2 Aug 2026
- 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning8 Jul 2026
- 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning30 Jun 2026
- 2026Learning Where to Look: A Reinforcement Learning Framework for Robust Micro-Ultrasound Prostate Cancer Detection29 Jun 2026
- 2026Scaling Participation in Modular AI Systems5 Jun 2026
- 2026CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations28 May 2026
- 2026Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals26 May 2026
- 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation26 May 2026
26 more
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis19 May 2026
- 2026Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classification17 May 2026
- 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents11 May 2026
- 2026Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis6 May 2026
- 2026CTM-AI: A Blueprint for General AI Inspired by a Model of Consciousness30 Apr 2026
- 2026A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification29 Apr 2026
- 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning9 Apr 2026
- 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound22 Mar 2026
- 2026AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning15 Mar 2026
- 2026When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains1 Mar 2026
- 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos1 Mar 2026
- 2026MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine1 Mar 2026
- 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video25 Feb 2026
- 2026Visual Memory Injection Attacks for Multi-Turn Conversations17 Feb 2026
- 2026ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation4 Feb 2026
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergy8 Jan 2026
- 2026MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics5 Jan 2026
- 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning28 Nov 2025
- 2025The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality22 Nov 2025
- 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting31 Oct 2025
- 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning22 Oct 2025
- 2025Automated Structured Radiology Report Generation with Rich Clinical Context1 Oct 2025
- 2025UniECG: Understanding and Generating ECG in One Unified Model23 Sep 2025
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning22 Aug 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun 2025
- 2025Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs16 Jun 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
- 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoningcited by 6
- 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learningcited by 5
- 2025Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMscited by 3
- 2026When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gainscited by 3
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
- 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 2
- 2025The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Qualitycited by 1
- 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Videocited by 1
- 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
- 2026AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoningcited by 0
12 most cited of 14 All 14 (JSON)
- 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning8 Jul 2026
- 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning30 Jun 2026
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis19 May 2026
- 2026Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classification17 May 2026
- 2026Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis6 May 2026
- 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning9 Apr 2026
- 2026AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning15 Mar 2026
- 2026When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains1 Mar 2026
- 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video25 Feb 2026
- 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning28 Nov 2025
- 2025The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality22 Nov 2025
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning22 Aug 2025
12 newest of 14 All 14 (JSON)
- 2026Visual Memory Injection Attacks for Multi-Turn Conversationscited by 2
- 2026MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicinecited by 0
- 2026CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluationscited by 0
- 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0
- 2026Volumetric Radiology AI in the Era of Multimodal Large Language Models20 Aug 2026
- 2026CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations28 May 2026
- 2026MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine1 Mar 2026
- 2026Visual Memory Injection Attacks for Multi-Turn Conversations17 Feb 2026
- 2026ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretationcited by 9
- 2026MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamicscited by 7
- 2025UniECG: Understanding and Generating ECG in One Unified Modelcited by 4
- 2025PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reportingcited by 4
- 2025Automated Structured Radiology Report Generation with Rich Clinical Contextcited by 2
- 2026CTM-AI: A Blueprint for General AI Inspired by a Model of Consciousnesscited by 2
- 2026Scaling Participation in Modular AI Systemscited by 2
- 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videoscited by 1
- 2026A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classificationcited by 1
- 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
- 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generationcited by 1
- 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoningcited by 0
12 most cited of 19 All 19 (JSON)
- 2026TRACE: Expert-Aligned ECG Representation Learning with Rigorous Benchmarking and Real-World Validation in Acute Cardiac Care28 Sep 2026
- 2026Continuous-Latent Predictive Modeling with Semantic Alignment for EEG-Language Foundation Models12 Aug 2026
- 2026MedUPS: Towards Diagnostic Assistance in Uncommon Medical Cases with Large Language Models2 Aug 2026
- 2026Learning Where to Look: A Reinforcement Learning Framework for Robust Micro-Ultrasound Prostate Cancer Detection29 Jun 2026
- 2026Scaling Participation in Modular AI Systems5 Jun 2026
- 2026Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals26 May 2026
- 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation26 May 2026
- 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents11 May 2026
- 2026CTM-AI: A Blueprint for General AI Inspired by a Model of Consciousness30 Apr 2026
- 2026A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification29 Apr 2026
- 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound22 Mar 2026
- 2026Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos1 Mar 2026
12 newest of 19 All 19 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2026MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcement
- 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images
- 2025MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoom
- 2025Toward Effective Reinforcement Learning Fine-Tuning for Medical VQA in Vision-Language Models
- 2025Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimization
- 2025Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patients
- 2025InfiMed: Low-Resource Medical MLLMs with Advancing Understanding and Reasoning
- 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding