mlsanityData to 14 Oct 2026

Graph-Text Multi-Modal Pre-training for Medical Representation Learning

arXiv 2203.09994 ↗First posted 18 Mar 2022cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

As the volume of Electronic Health Records (EHR) sharply grows, there has been emerging interest in learning the representation of EHR for healthcare applications. Representation learning of EHR requires appropriate modeling of the two dominant modalities in EHR: structured data and unstructured text. In this paper, we present MedGTX, a pre-trained model for multi-modal representation learning of the structured and textual EHR data. MedGTX uses a novel graph encoder to exploit the graphical nature of structured EHR data, and a text encoder to handle unstructured text, and a cross-modal encoder to learn a joint representation space. We pre-train our model through four proxy tasks on MIMIC-III, an open-source EHR data, and evaluate our model on two clinical benchmarks and three novel downstream tasks which tackle real-world problems in EHR data. The results consistently show the effectiveness of pre-training the model for joint representation of both structured and unstructured information from EHR. Given the promising performance of MedGTX, we believe this work opens a new door to jointly understanding the two fundamental modalities of EHR data.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

28 of 51 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Graph Attention Networkscited by 8,923
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
16 more
  1. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  2. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  3. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  4. 2016GRAM: Graph-based Attention Model for Healthcare Representation Learningcited by 156
  5. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  6. 2016De-identification of Patient Notes with Recurrent Neural Networkscited by 88
  7. 2019Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformercited by 88
  8. 2018MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcarecited by 72
  9. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  10. 2016Multi-layer Representation Learning for Medical Conceptscited by 69
  11. 2016Deepr: A Convolutional Net for Medical Recordscited by 60
  12. 2020Knowledge Graph-based Question Answering with Electronic Health Recordscited by 19
  13. 2016SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptionscited by 18
  14. 2016Knowledge Graph Representation with Jointly Structural and Textual Encodingcited by 17
  15. 2019mathttMedGraph: Structural and Temporal Representation Learning of Electronic Medical Recordscited by 10
  16. 2018Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansioncited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 6; 2025: 2.

10 papers.

  1. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 17
  2. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  3. 2022Unsupervised pre-training of graph transformers on patient population graphscited by 9
  4. 2024CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysiscited by 6
  5. 2024Bridging Stepwise Lab-Informed Pretraining and Knowledge-Guided Learning for Diagnostic Reasoningcited by 3
  6. 2023Multi-modal Graph Learning over UMLS Knowledge Graphscited by 2
  7. 2025CAST: Cross Attention based multimodal fusion of Structure and Text for materials property predictioncited by 2
  8. 2024From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHRcited by 1
  9. 2024MPLite: Multi-Aspect Pretraining for Mining Clinical Health Recordscited by 1
  10. 2025Patient-level Information Extraction by Consistent Integration of Textual and Tabular Evidence with Bayesian Networkscited by 0