# Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs

arXiv: https://arxiv.org/abs/2212.08072 · HTML: https://mlsanity.com/paper/2212.08072/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 13 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit](https://mlsanity.com/paper/2010.01165/) (2020)
- [Estimating Redundancy in Clinical Text](https://mlsanity.com/paper/2105.11832/) (2021)
- [MedCATTrainer: A Biomedical Free Text Annotation Interface with Active Learning and Research Use Case Specific Customisation](https://mlsanity.com/paper/1907.07322/) (2019)

## Built on by (most cited first, 11 of 11)

- [Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI](https://mlsanity.com/paper/2311.01463/) (2023)
- [HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responses](https://mlsanity.com/paper/2312.15883/) (2023)
- [CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines](https://mlsanity.com/paper/2402.04400/) (2024)
- [ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes](https://mlsanity.com/paper/2403.05795/) (2024)
- [Can GPT-3.5 Generate and Code Discharge Summaries?](https://mlsanity.com/paper/2401.13512/) (2024)
- [Large Language Models for Medical Forecasting -- Foresight 2](https://mlsanity.com/paper/2412.10848/) (2024)
- [CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2509.03643/) (2025)
- [Foundation Models for Clinical Records at Health System Scale](https://mlsanity.com/paper/2507.00574/) (2025)
- [Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction](https://mlsanity.com/paper/2603.24562/) (2026)
- [Scaling Electronic Health Record Foundation Models for Population Health Management](https://mlsanity.com/paper/2506.00209/) (2025)
- [Building Models of Neurological Language](https://mlsanity.com/paper/2506.06208/) (2025)

## Abstract (arXiv)

Background: Electronic Health Records hold detailed longitudinal information about each patient's health status and general clinical history, a large portion of which is stored within the unstructured text. Existing approaches focus mostly on structured data and a subset of single-domain outcomes. We explore how temporal modelling of patients from free text and structured data, using deep generative transformers can be used to forecast a wide range of future disorders, substances, procedures or findings. Methods: We present Foresight, a novel transformer-based pipeline that uses named entity recognition and linking tools to convert document text into structured, coded concepts, followed by providing probabilistic forecasts for future medical events such as disorders, substances, procedures and findings. We processed the entire free-text portion from three different hospital datasets totalling 811336 patients covering both physical and mental health. Findings: On tests in two UK hospitals (King's College Hospital, South London and Maudsley) and the US MIMIC-III dataset precision@10 0.68, 0.76 and 0.88 was achieved for forecasting the next disorder in a patient timeline, while precision@10 of 0.80, 0.81 and 0.91 was achieved for forecasting the next biomedical concept. Foresight was also validated on 34 synthetic patient timelines by five clinicians and achieved relevancy of 97% for the top forecasted candidate disorder. As a generative model, it can forecast follow-on biomedical concepts for as many steps as required. Interpretation: Foresight is a general-purpose model for biomedical concept modelling that can be used for real-world risk forecasting, virtual trials and clinical research to study the progression of disorders, simulate interventions and counterfactuals, and educational purposes.