# CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records

arXiv: https://arxiv.org/abs/2506.15118 · HTML: https://mlsanity.com/paper/2506.15118/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 64 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences](https://mlsanity.com/paper/2306.07906/) (2023)
- [Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer](https://mlsanity.com/paper/1906.04716/) (2019)
- [ML-Decoder: Scalable and Versatile Classification Head](https://mlsanity.com/paper/2111.12933/) (2021)
- [Pre-trained Summarization Distillation](https://mlsanity.com/paper/2010.13002/) (2020)
- [GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs](https://mlsanity.com/paper/2305.12788/) (2023)
- [A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine](https://mlsanity.com/paper/2405.08603/) (2024)
- [Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration](https://mlsanity.com/paper/2102.04110/) (2021)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2504.14772/) (2025)
- [RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records](https://mlsanity.com/paper/2403.00815/) (2024)
- [Literature-Augmented Clinical Outcome Prediction](https://mlsanity.com/paper/2111.08374/) (2021)
- [Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation](https://mlsanity.com/paper/2402.18150/) (2024)
- [Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Models](https://mlsanity.com/paper/2307.04910/) (2023)
- [Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models](https://mlsanity.com/paper/2411.07611/) (2024)
- [Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers](https://mlsanity.com/paper/2212.00587/) (2022)

## Abstract (arXiv)

Electronic Health Records (EHR)-based disease prediction models have demonstrated significant clinical value in promoting precision medicine and enabling early intervention. However, existing large language models face two major challenges: insufficient representation of medical knowledge and low efficiency in clinical deployment. To address these challenges, this study proposes the CKD-EHR (Clinical Knowledge Distillation for EHR) framework, which achieves efficient and accurate disease risk prediction through knowledge distillation techniques. Specifically, the large language model Qwen2.5-7B is first fine-tuned on medical knowledge-enhanced data to serve as the teacher model.It then generates interpretable soft labels through a multi-granularity attention distillation mechanism. Finally, the distilled knowledge is transferred to a lightweight BERT student model. Experimental results show that on the MIMIC-III dataset, CKD-EHR significantly outperforms the baseline model:diagnostic accuracy is increased by 9%, F1-score is improved by 27%, and a 22.2 times inference speedup is achieved. This innovative solution not only greatly improves resource utilization efficiency but also significantly enhances the accuracy and timeliness of diagnosis, providing a practical technical approach for resource optimization in clinical settings. The code and data for this research are available athttps://github.com/209506702/CKD_EHR.