# Pre-training of Graph Augmented Transformers for Medication Recommendation

arXiv: https://arxiv.org/abs/1906.00346 · HTML: https://mlsanity.com/paper/1906.00346/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2019.

## Built on by (most cited first, 30 of 127)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Trustworthy Graph Neural Networks: Aspects, Methods and Trends](https://mlsanity.com/paper/2205.07424/) (2022)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data](https://mlsanity.com/paper/2306.02664/) (2023)
- [MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph](https://mlsanity.com/paper/2106.04509/) (2021)
- [GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs](https://mlsanity.com/paper/2305.12788/) (2023)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinations](https://mlsanity.com/paper/2105.02711/) (2021)
- [Molecular Contrastive Learning with Chemical Element Knowledge Graph](https://mlsanity.com/paper/2112.00544/) (2021)
- [LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation](https://mlsanity.com/paper/2405.20646/) (2024)
- [Change Matters: Medication Change Prediction with Recurrent Residual Networks](https://mlsanity.com/paper/2105.01876/) (2021)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Collaborative Graph Learning with Auxiliary Text for Temporal Event Prediction in Healthcare](https://mlsanity.com/paper/2105.07542/) (2021)
- [Large Language Model Distilling Medication Recommendation Model](https://mlsanity.com/paper/2402.02803/) (2024)
- [MedGPT: Medical Concept Prediction from Clinical Narratives](https://mlsanity.com/paper/2107.03134/) (2021)
- [Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embedding](https://mlsanity.com/paper/2108.03625/) (2021)
- [Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs](https://mlsanity.com/paper/2112.05195/) (2021)
- [Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review](https://mlsanity.com/paper/2303.06471/) (2023)
- [Embedding in Recommender Systems: A Survey](https://mlsanity.com/paper/2310.18608/) (2023)
- [CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT](https://mlsanity.com/paper/2404.15201/) (2024)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Defending Graph Convolutional Networks against Dynamic Graph Perturbations via Bayesian Self-supervision](https://mlsanity.com/paper/2203.03762/) (2022)
- [BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients](https://mlsanity.com/paper/2103.10928/) (2021)
- [Self-Supervised Graph Learning with Hyperbolic Embedding for Temporal Health Event Prediction](https://mlsanity.com/paper/2106.04751/) (2021)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)

## Abstract (arXiv)

Medication recommendation is an important healthcare application. It is commonly formulated as a temporal prediction task. Hence, most existing works only utilize longitudinal electronic health records (EHRs) from a small number of patients with multiple visits ignoring a large number of patients with a single visit (selection bias). Moreover, important hierarchical knowledge such as diagnosis hierarchy is not leveraged in the representation learning process. To address these challenges, we propose G-BERT, a new model to combine the power of Graph Neural Networks (GNNs) and BERT (Bidirectional Encoder Representations from Transformers) for medical code representation and medication recommendation. We use GNNs to represent the internal hierarchical structures of medical codes. Then we integrate the GNN representation into a transformer-based visit encoder and pre-train it on EHR data from patients only with a single visit. The pre-trained visit encoder and representation are then fine-tuned for downstream predictive tasks on longitudinal EHRs from patients with multiple visits. G-BERT is the first to bring the language model pre-training schema into the healthcare domain and it achieved state-of-the-art performance on the medication recommendation task.