# Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

arXiv: https://arxiv.org/abs/2407.11536 · HTML: https://mlsanity.com/paper/2407.11536/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 11 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese](https://mlsanity.com/paper/2309.04175/) (2023)
- [IvyGPT: InteractiVe Chinese pathwaY language model in medical domain](https://mlsanity.com/paper/2307.10512/) (2023)

## Built on by (most cited first, 4 of 4)

- [Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation](https://mlsanity.com/paper/2505.24174/) (2025)
- [Continual Pre-Training is (not) What You Need in Domain Adaption](https://mlsanity.com/paper/2504.13603/) (2025)
- [KRAL: Knowledge and Reasoning Augmented Learning for LLM-assisted Clinical Antimicrobial Therapy](https://mlsanity.com/paper/2511.15974/) (2025)
- [Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset](https://mlsanity.com/paper/2603.24772/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q&A data exhibit extraordinary disease diagnostic abilities. However, we observed that despite improvements in specific domain knowledge, the performance of medical LLM in long-context understanding has significantly declined, especially compared to general language models with similar parameters. The purpose of this study is to investigate the phenomenon of reduced performance in understanding long-context in medical LLM. We designed a series of experiments to conduct open-book professional knowledge exams on all models to evaluate their ability to read long-context. By adjusting the proportion and quantity of general data and medical data in the process of fine-tuning, we can determine the best data composition to optimize the professional model and achieve a balance between long-context performance and specific domain knowledge.