mlsanityData to 14 Oct 2026

Unsupervised pre-training of graph transformers on patient population graphs

arXiv 2207.10603 ↗First posted 21 Jul 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Pre-training has shown success in different areas of machine learning, such as Computer Vision, Natural Language Processing (NLP), and medical imaging. However, it has not been fully explored for clinical data analysis. An immense amount of clinical records are recorded, but still, data and labels can be scarce for data collected in small hospitals or dealing with rare diseases. In such scenarios, pre-training on a larger set of unlabelled clinical data could improve performance. In this paper, we propose novel unsupervised pre-training techniques designed for heterogeneous, multi-modal clinical data for patient outcome prediction inspired by masked language modeling (MLM), by leveraging graph deep learning over population graphs. To this end, we further propose a graph-transformer-based network, designed to handle heterogeneous clinical data. By combining masking-based pre-training with a transformer-based network, we translate the success of masking-based pre-training in other domains to heterogeneous clinical data. We show the benefit of our pre-training method in a self-supervised and a transfer learning setting, utilizing three medical datasets TADPOLE, MIMIC-III, and a Sepsis Prediction Dataset. We find that our proposed pre-training methods help in modeling the data at a patient and population level and improve performance in different fine-tuning tasks on all datasets.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 50 of its 82 references that are in the map. First posted on arXiv 21 Jul 2022.

Builds on

50 of 82 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2017Graph Attention Networkscited by 8,923
  7. 2018How Powerful are Graph Neural Networks?cited by 4,393
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
38 more
  1. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  2. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  3. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  4. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  6. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  7. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  8. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  9. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  10. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  11. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  12. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 177
  13. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  14. 2020Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotationcited by 128
  15. 2019Pre-training of Graph Augmented Transformers for Medication Recommendationcited by 127
  16. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 122
  17. 2018Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Diseasecited by 99
  18. 2017Unsupervised Learning by Predicting Noisecited by 98
  19. 2020Differentiable Graph Module (DGM) for Graph Convolutional Networkscited by 86
  20. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 76
  21. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 71
  22. 2019Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Predictioncited by 70
  23. 2021Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Futurecited by 67
  24. 2017TimeNet: Pre-trained deep recurrent neural network for time series classificationcited by 46
  25. 2019InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Predictioncited by 39
  26. 2020Deep Representation Learning of Electronic Health Records to Unlock Patient Stratification at Scalecited by 39
  27. 2020Latent-Graph Learning for Disease Predictioncited by 39
  28. 2020Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognitioncited by 35
  29. 2020Edge-variational Graph Convolutional Networks for Uncertainty-aware Disease Predictioncited by 25
  30. 2019Transfer Learning for Clinical Time Series Analysis using Deep Neural Networkscited by 22
  31. 2020Self-Supervised Graph Transformer on Large-Scale Molecular Datacited by 17
  32. 2021RA-GCN: Graph Convolutional Network for Disease Prediction Problems with Imbalanced Datacited by 14
  33. 2018Self-Attention Equipped Graph Convolutions for Disease Predictioncited by 11
  34. 2021IA-GCN: Interpretable Attention based Graph Convolutional Network for Disease predictioncited by 10
  35. 2022Graph-Text Multi-Modal Pre-training for Medical Representation Learningcited by 10
  36. 2021Leveraging Time Irreversibility with Order-Contrastive Pre-trainingcited by 7
  37. 2018Multi-modal Disease Classification in Incomplete Datasets Using Geometric Matrix Completioncited by 5
  38. 2019Learning Universal Graph Neural Network Embeddings With Aid Of Transfer Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 2.

9 papers.

  1. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  2. 2024Graph Transformers: A Surveycited by 32
  3. 2024MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Predictioncited by 9
  4. 2024FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Predictioncited by 6
  5. 2024Bridging Data Gaps in Healthcare: A Scoping Review of Transfer Learning in Biomedical Data Analysiscited by 6
  6. 2024Fusing Echocardiography Images and Medical Records for Continuous Patient Stratificationcited by 3
  7. 2025EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Recordscited by 3
  8. 2026DT-BEHRT: Disease Trajectory-aware Transformer for Interpretable Patient Representation Learningcited by 1
  9. 2026MRC-GAT: A Meta-Relational Copula-Based Graph Attention Network for Interpretable Multimodal Alzheimer's Disease Diagnosiscited by 0