# Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimization

arXiv: https://arxiv.org/abs/2409.12741 · HTML: https://mlsanity.com/paper/2409.12741/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 7 of 7)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective](https://mlsanity.com/paper/2404.04626/) (2024)
- [Large Language Models to Identify Social Determinants of Health in Electronic Health Records](https://mlsanity.com/paper/2308.06354/) (2023)
- [Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks](https://mlsanity.com/paper/2404.14723/) (2024)

## Built on by (most cited first, 6 of 6)

- [The Rise of Small Language Models in Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2504.17119/) (2025)
- [Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization](https://mlsanity.com/paper/2503.00018/) (2025)
- [Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching](https://mlsanity.com/paper/2507.04099/) (2025)
- [Deep Learning-based Prediction of Clinical Trial Enrollment with Uncertainty Estimates](https://mlsanity.com/paper/2507.23607/) (2025)
- [FocusMed: A Large Language Model-based Framework for Enhancing Medical Question Summarization with Focus Identification](https://mlsanity.com/paper/2510.04671/) (2025)
- [Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation](https://mlsanity.com/paper/2608.20794/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) fine tuning is underutilized in the field of medicine. Two of the most common methods of fine tuning are Supervised Fine Tuning (SFT) and Direct Preference Optimization (DPO), but there is little guidance informing users when to use either technique. In this investigation, we compare the performance of SFT and DPO for five common natural language tasks in medicine: Classification with text data, Classification with numeric data, Clinical Reasoning, Summarization, and Clinical Triage. We find that SFT alone is sufficient for Classification with text data, whereas DPO improves performance for the more complex tasks of Clinical Reasoning, Summarization and Clinical Triage. Our results establish the role and importance of DPO fine tuning within medicine, and consequently call attention to current software gaps that prevent widespread deployment of this technique.