mlsanityData to 7 Oct 2026

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

arXiv 2609.40361 ↗First posted 30 Sep 2026cs.LG cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores matrix with one row per evaluation instance and one column per candidate prompt; cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the prompt evolution search reads from - the scores matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via vision-encoder-tuned SFT or medical pretraining - is a prerequisite that prompt search cannot replace - our recipe extends reflective prompt evolution from text-only data to multimodal clinical decision-making.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 34 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

21 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging5 Jan
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 18

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  10. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  11. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  12. 2025MedGemma Technical Reportcited by 398
6 more
  1. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  2. 2023Towards Generalist Biomedical AIcited by 286
  3. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  4. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
  5. 2020Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classificationcited by 84
  6. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models
  2. 2026When Prompts Mislead: Textual Dominance and Diagnostic Bias in MLLMs
  3. 2026RLMOpt: Adaptive Prompt Optimization via Recursive Language Models
  4. 2025Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
  5. 2025Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies
  6. 2026MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
  7. 2025Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications
  8. 2026Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation