mlsanityData to 7 Oct 2026

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation

arXiv 2406.06496 ↗First posted 10 Jun 2024cs.LG cs.CL cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Recent advances in generative vision-language models (VLMs) have exciting potential implications for AI in radiology, yet VLMs are also known to produce hallucinations, nonsensical text, and other unwanted behaviors that can waste clinicians' time and cause patient harm. Drawing on recent work on direct preference optimization (DPO), we propose a simple method for modifying the behavior of pretrained VLMs performing radiology report generation by suppressing unwanted types of generations. We apply our method to the prevention of hallucinations of prior exams, addressing a long-established problem behavior in models performing chest X-ray report generation. Across our experiments, we find that DPO fine-tuning achieves a 3.2-4.8x reduction in lines hallucinating prior exams while maintaining model performance on clinical accuracy metrics. Our work is, to the best of our knowledge, the first work to apply DPO to medical VLMs, providing a data- and compute- efficient way to suppress problem behaviors while maintaining overall clinical accuracy.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 29 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

17 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement11 Mar

Other references · 16

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  6. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  7. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  8. 2024Direct Preference Optimization with an Offsetcited by 122
  9. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  10. 2022Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priorscited by 48
  11. 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
  12. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
4 more
  1. 2023Style-Aware Radiology Report Generation with RadGraph and Few-Shot Promptingcited by 28
  2. 2023Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT modelscited by 27
  3. 2024Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedbackcited by 9
  4. 2024NOTE: Notable generation Of patient Text summaries through Efficient approach based on direct preference optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  2. 2024ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reportscited by 9
  3. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  4. 2025Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentationcited by 1
  5. 2025Benchmark Success, Clinical Failure: When Reinforcement Learning Optimizes for Benchmarks, Not Patientscited by 0
  6. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  7. 2026Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Random Direct Preference Optimization for Radiography Report Generation
  2. 2026Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation
  3. 2026Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering
  4. 2025RadHiera: Semantic Hierarchical Reinforcement Learning for Medical Report Generation
  5. 2025Process Reward Models for Sentence-Level Verification of LVLM Radiology Reports
  6. 2026Enhancing Visual Reasoning in Chest X-Ray Report Generation Using Reinforcement Learning
  7. 2025Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learning
  8. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale