mlsanityData to 14 Oct 2026

CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records

arXiv 2506.15118 ↗First posted 18 Jun 2025cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Electronic Health Records (EHR)-based disease prediction models have demonstrated significant clinical value in promoting precision medicine and enabling early intervention. However, existing large language models face two major challenges: insufficient representation of medical knowledge and low efficiency in clinical deployment. To address these challenges, this study proposes the CKD-EHR (Clinical Knowledge Distillation for EHR) framework, which achieves efficient and accurate disease risk prediction through knowledge distillation techniques. Specifically, the large language model Qwen2.5-7B is first fine-tuned on medical knowledge-enhanced data to serve as the teacher model.It then generates interpretable soft labels through a multi-granularity attention distillation mechanism. Finally, the distilled knowledge is transferred to a lightweight BERT student model. Experimental results show that on the MIMIC-III dataset, CKD-EHR significantly outperforms the baseline model:diagnostic accuracy is increased by 9%, F1-score is improved by 27%, and a 22.2 times inference speedup is achieved. This innovative solution not only greatly improves resource utilization efficiency but also significantly enhances the accuracy and timeliness of diagnosis, providing a practical technical approach for resource optimization in clinical settings. The code and data for this research are available athttps://github.com/209506702/CKD_EHR.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 26 of its 64 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

26 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions20 Apr
  2. 2024A Survey on Knowledge Distillation of Large Language Models20 Feb
  3. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models6 Feb

Other references · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  5. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  6. 2017Multitask learning and benchmarking with clinical time series datacited by 290
  7. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  8. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  9. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  10. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  11. 2019Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformercited by 88
  12. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 79
11 more
  1. 2020Pre-trained Summarization Distillationcited by 71
  2. 2023GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphscited by 58
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  4. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 44
  5. 2023Automated Paper Screening for Clinical Reviews Using Large Language Modelscited by 29
  6. 2024RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Recordscited by 21
  7. 2021Literature-Augmented Clinical Outcome Predictioncited by 20
  8. 2024Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generationcited by 19
  9. 2023Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Modelscited by 8
  10. 2024Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Modelscited by 4
  11. 2022Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformerscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Distilling Large Language Models for Efficient Clinical Information Extraction
  2. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study
  3. 2025Knowledge Graph Augmented Large Language Models for Disease Prediction
  4. 2023Lightweight Transformers for Clinical Natural Language Processing
  5. 2025TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning
  6. 2025Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
  7. 2025DR.EHR: Dense Retrieval for Electronic Health Record with Knowledge Injection and Synthetic Data
  8. 2025EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis