# Limits of Generative Pre-Training in Structured EMR Trajectories with Irregular Sampling

arXiv: https://arxiv.org/abs/2510.22878 · HTML: https://mlsanity.com/paper/2510.22878/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions](https://mlsanity.com/paper/2002.03478/) (2020)
- [Generating Synthetic Clinical Data that Capture Class Imbalanced Distributions with Generative Adversarial Networks: Example using Antiretroviral Therapy for HIV](https://mlsanity.com/paper/2208.08655/) (2022)

## Built on by (most cited first, 1 of 1)

- [PRIME-CVD: A Parametrically Rendered Informatics Medical Environment for Education in Cardiovascular Risk Modelling](https://mlsanity.com/paper/2603.19299/) (2026)

## Abstract (arXiv)

Foundation models refer to architectures trained on vast datasets using autoregressive pre-training from natural language processing to capture intricate patterns and motifs. They were originally developed to transfer such learned knowledge to downstream predictive tasks. Recently, however, some studies repurpose these learned representations for phenotype discovery without rigorous validation, risking superficially realistic but clinically incoherent embeddings. To test this mismatch, we trained two autoregressive models -- a sequence-to-sequence LSTM and a reduced Transformer -- on longitudinal ART for HIV and Acute Hypotension datasets. Controlled irregularity was added during training via random inter-visit gaps, while test sequences stayed complete. Patient-trajectory synthesis evaluated distributional and correlational fidelity. Both reproduced feature distributions but failed to preserve cross-feature structure -- showing that generative pre-training yields local realism but limited clinical coherence. These results highlight the need for domain-specific evaluation and support trajectory synthesis as a practical probe before fine-tuning or deployment.